back
99 comments
Just threw $100 your way.

I'm a big believer in this type of model and having you be a Django core contributor and South developer makes it a no brainer. Paying for quality open source is always a win in my book.

Godspeed!

I am, on the other hand, not a fan of this investment model. Indeed I would go so far as to say I don't think it works nor does it make much sense. Especially in the open source world.

I've been using django since 2006. This has been a suggestion since then. Why hasn't it been implemented if it is so necessary? It doesn't require a massive core shift, it doesn't even require that much work it seems? So why hasn't it been done with so many eyes on so well documented a platform on so accessible a language with so many commercial deployments? If no one has scratched this itch then paying for someone to scratch an itch you may think you are go

Will the person be deprived of their normal source of income for exactly the time period required to do the work? How is that justified as a cost? Maybe they are a contractor who will take the time out. If that is the case how can there be expandable requirements (longer goal objectives for more money? It could be this whole thing sits very well with their day to day job, but in the western world that is unlikely. I'm sure all intentions are good but it also strikes me as a bit questionable to do these things when lots of other people have committed much more centrally to the project with little immediate remuneration.

Thanks a lot - your contribution is much appreciated! I hope this helps set a trend.
Worth mentioning that if someone wants to ask me (I'm running this) questions about this I'll be checking here!
What are the problems with South, as you point to? I think those are a big part of the selling point for your project. :)

Additionally, could you explain how the project will not be obviated by Django's plans to implement built-in schema migration in, I think, 1.6 or 1.7?

You taken on one of the most important aspects of managing Django services, so I'm genuinely interested, but I just want to know what the thinking is behind this.

Thanks!

You say "Schema migration with Django has had a long and complex history". Can you explain, even briefly, why migrations haven't been a part of Django's core so far? I was under the impression that it was a deliberate design choice. What made you change your mind now?

As of now you have collected 10k GBP, what are you going to do with the money with the 3k+ in unplanned additional funds? Are you going to adjust you hourly rate (not that I mind your initial "quote" wasn't that high to begin with), donate to charity, deliver more code/value, buy the Django core devs a beer? Genuinely curious. And congrats on making your goal, I love this model and I love that you had the courage to make this effort. Succeeding with this campaign publicly shows that contributing to open source software has to be an unpaid endeavor (yes I know that there are employees who get paid to contribute to open software). I wouldn't mind if more talented devs sought out crowdfunding to enhance OSS.
Awesome. I will make a donation if I can get approval.

How will the new rebase command be different from truncating the migration table and regenerating initial migrations?

Are non-django field types like (postgresql hstore) going to have some love as part of these migrations? If not, a proper and standardized way to write plugins to extend it would be awesome in order to be able to migrate "nonstandard" field types.
Is there a reason to include this directly in Django rather than leaving it as a pluggable app? Updates/bug-fixes/features are way easier to push when it's not tied into the official update cycle, letting you be a lot more agile.
Congratulations - you've just passed £3500!
Does this mean anything will happen with Oracle?

I know that I'm perhaps the odd man out, but South just doesn't work reliably on Oracle, and I really, really miss it.

Alternately, if I were to donate a running RDS instance with Oracle on it, would that help in any way? I seem to remember an issue (that affects me) reported, and effectively disregarded with a message similar to, but not necessarily that you don't have Oracle, so it wasn't likely to get fixed.

I haven't used the Django ORM in over a year and I'm pretty happy with Flask + SQLAlchemy for the moment. That said, I instantly backed the campaign when I saw it. South has been a tremendously helpful tool I used in virtually all my Django projects since its inception and I'm glad I can finally give something back.
Hi Andrew, congrats on the campaign and reaching your goal so quickly! And thank you for South! I've been using South for some years and it would be impossible to imagine working on a Django project wihout it. Two quick questions:

- Are there any South "Good practices" books which you would recommend? South's documentation is great, but it might be interesting to look at some real-world project examples to learn a few new tricks.

- I'm interested in particular in data migration development practices. How do other developers go about developing and testing them? I've discovered that by raising an Exception during the data migration, South will rollback the specific migration, allowing me to iteratively test/debug the migration until it seems bug-free and ready. Are there any other ways to do it (would running the data migration with the "fake" option allow me full read access to the database, but would rollback / skip any writes?).

Thank you for your time!

There was an interesting chat-log[1] someone posted on /r/Django on reddit talking about dealing with South migration merging/conflicts, and other issues that crop up in a multi-person project.

I'm curious about the best practices for data-migrations as well; I've seen a few places[2][3] suggest that wrapping your fixture loading "./manage.py loaddata <fixture>" in datamigrations is better than leaving it up to initial_data.{yaml,json}

Edit: here's the one I was thinking of: South common pitfalls: http://andrewingram.net/2012/dec/common-pitfalls-django-sout...

[1] https://gist.github.com/sjl/4438002

[2] http://djangosnippets.org/snippets/2897/

[3] http://stackoverflow.com/questions/5472925/django-loading-da...

Hi Greg,

There's no books or anything, alas, nor is there a very good corpus of material on this field - there's a few good talks out there, but I've not seen a great deal of written material.

Perhaps I need to write a book as well? Not sure I need another Kickstarter, though!

I would checkout Scala-Migrations for guidance. https://code.google.com/p/scala-migrations/
When I saw this at the top of HN I was thinking "Damn it! The South guy already has a plan for this!" ... then I see it IS the South guy asking for funding to implement his plan!

Done!

Oh, and thanks for the email support now and then :-)

I don't know whether you'll find it useful, but when I wrote a migrations add-on for Rails, I spent a lot of time learning how to query Postgres and MySQL for the status of current foreign keys, indexes, and "check" constraints, including special extensions like "where" clauses on indexes. If you need to do something similar, perhaps you can borrow the queries from here:

    https://github.com/pjungwir/db_leftovers
If that seems useful and you have any questions, please let me know!
Having South take that leap into Django core will be awesome.

I'll bet one natural result of it entering core will be better shared understanding of best practices & foibles of different backends.

Managing migrations on increasingly large tables is stressful for me primarily due to my lack of knowledge, moreso than fragility in any piece of the puzzle.

In any case, I've thrown in my pledge - best of luck!

(If anyone has any good recommended reading for DBA-type knowledge in a devops world, I'd love an Amazon link you think is worthwhile.)

I was hesitant to choose Python 3.3 for a new project with Django 1.5 because of compatibility issues with add-on libraries. I've been able to get South working from the tip of the hg repo, which was a big one for me.

Thanks for working on this, it makes me much more comfortable with my Python 3.3 decision knowing migrations will have first class support in Django. I just sent in my contribution.

I'm really glad to see someone tackling this; even better that it's someone with plenty of experience and insight. I wish more people would give me opportunities to help fund big features like this in open source.
I found myself talking to a co-worker about this being a problem last week.

We found that having this functionality as part of the core is something that makes a framework feel significantly more agile. Virtually nobody gets their schema exactly right the first time. Asking developers to to learn about and use 3rd party software (if it even exists) adds a very real friction to migrations. Altering your data model from time to time is a natural operation that occurs during application development and should not seem out of place.

I'm very excited for this, thanks Andrew!

Andrew--

We use Django and South in our projects, and really appreciate the work you have done. I just contributed, and am excited about this project.

My one suggestion/request: Please make it an emphasis to create awesome documentation. I am familiar with South now at this point but often find myself wishing more use cases and more examples were documented. I can usually "figure it out," but great documentation saves everyone time. This seems like a great project, so make the documentation great too!

EDIT: By 'not possible' I mean it's not possible to delegate responsibility for it to a tool that derives what the differences in schema are.

Schema migrations are not possible in general.

Yes, you can diff two schemas and generate statements to turn one into the other. Adding columns works fine this way. Removing columns works fine.

Do you really need a library to add and remove columns? Not really. But rollback, someone will say. Schema migration libraries let you undo things. Nope. You can't rollback a drop column once it is committed, that data is gone.

What happens when you run into issues in production that you didn't have in your test/development environment?

Basically, automated schema migrations don't work. However, I guess if you haven't been through it you'll just have to learn the hard way. :)

SQL gives you a very flexible environment. You never have to break existing code to push out new features, if you are just a tiny bit careful.

Off-topic but how is money "earned" from Kickstarted taxed?
As someone who just spent a lot of time learning Django, with it being the first web framework I've ever used, thank you! The current system resulted in some serious headaches while learning, and these changes will definitely remove a huge barrier to entry for people like me.
Thanks for doing this Andrew - our company uses South daily, and I've griped about a lack of a core Django migration capability from day one. It's a pleasure to support such a specific and useful project. Money well spent.
How is this different from the Ruby on Rails way?

I have been away from the Django project for a long time, and I really like the Rails way. I have supported anyway, everyone deserves a proper migration system.

This is a no brainer donation in so many ways... from the obvious of helping you complete this which will benefit so many people directly, to helping establish this very fine precedence!

Thanks and good luck!

For stuff like this, why don't people just set up their own donation site? The target audience isn't necessarily off-put by other steps, and you can get past the kickstarter fees.
First thing I thought while reading this kickstarter was "Why re-invent South.. it's already perfect!". I changed my mind when I understood who the author was. To be honest, I don't fully understand why the new version is needed.. we should just include South in the Django core, problem solved. But hey, I'm no expert in schema migration and I'm happy to help the author moves forward with his project. The only complain I had with south was that it wasn't installed by default ;-)
Great idea! First time pledging to kickstarter. I think there are a lot of improvements to be made related to database migrations. I built DevJoist (devjoist.com) with a similar goal in mind, but primarily for PHP developers. Multiple people have pointed to Django south as a great solution, so I'm happy to hear you're going to make it even better. If you ever want to chat hit me up! Email in profile.
Speaking of migrations, I just got a super-simple migration system merged into Chicago Boss!

https://github.com/evanmiller/ChicagoBoss/blob/master/README...

Well, almost...it's still missing one pull. Even if it's very, very basic, I'm pretty happy to have something rather than nothing, though.

Mybatis migrations is also a great alternative for schema migrations

http://www.mybatis.org/migrations/index.html

It is made by the Mybatis team but it not dependent on the Mybatis object mapping framework or on Java. It runs from the command line.

Wow, I haven't been following Kickstarter projects as closely before, but this seems to be rising quite quickly!

I guess all the right ingredients are here: someone competent providing a useful service with an achievable goal. Nicely done and good luck!

So, it looks like this is going insanely well, as a kickstarter.

What do you plan to do with the excess money, if you raise more than the highest, £7000 amount? Donate the remainder to the Django Software Foundation, hopefully?

I love kickstarter as much as the next guy, but this is stupid. You just threw 10% of your donations away. I don't think Kickstarter is necessarily the best tool for this kind of fundraising.
I just backed this Kickstarter, good luck on hitting your stretch goals. While South is decent enough, I've always thought it needed replacement or a lot of renovation.
This looks awesome. One question, why not release it as a third party module first and then merge it into the core later ala staticfiles?
wow! More than 10.000 pounds. Are you creating a new way to support Open Source software? Hope this catch on for other projects.
I hope someone can invent an SQL diff engine, e.g. input two files of creat table sql and output the alter table sql.
this is great. i'm planning on contributing to this and will try and convince some of my colleagues to do so as well. South is very handy and much appreciated but undoubtedly it can be improved alot and should be part of django core. This project will be wonderfully useful to the whole community.
What is bizarre is I saw it, thought it was great, came to post it on this site, and boom, was far too late.
What about using SQLAlchemy Migrations?
Just wanted to say thanks for South, it makes my projects a whole lot easier. Looking forward to this.
Thank you, this was absolutely necessary. Can't wait for it to release.
Some stats on the traffic sources to this page. Looks like Twitter was important to getting this project funded:

https://bitly.com/ZhmXuw+

How about this:

- I use South which is pretty good

- I use a NoSQL db. I will lose some things, but gain a lot as well (and in my point of view, the gains are bigger)

I'm not sure 'buying features' of software is a good way to go. (I'm not saying it's bad)

Edit: Why the hate? It certainly is better to have an integrated solution, I'm not saying this proposal is bad, only that there are other alternatives.

And the point is moot, donations went over the target already, hopefully this is going to be something big!