Upgrade to Django 1.9

This is a fairly extensive commit, since most of the requirements had
to be upgraded as well. Dependencies have been locked down to specific
versions for consistency and troubleshooting. Notable changes:

  - Moved wsgi.py to a more standard location, updated apache config.
    Now referenced in roundware.settings.WSGI_APPLICATION

  - For roundware.api2, implemented the new Application Configuration
    structure. See http://stackoverflow.com/q/32795227/1943591

  - django-guardian now creates its own AnonymousUser in the database.
    Removed AnonymousUser references from fixtures to avoid conflicts.
    http://django-guardian.readthedocs.io/en/stable/configuration.html

  - django-guardian assign is being depricated for assign_perm
    See django-guardian/django-guardian@1419048

  - ManyToManyField no longer accepts null=True

  - django.contrib.auth.models.User should not be accessed directly.
    Use django.conf.settings.AUTH_USER_MODEL instead

  - Django changed the way it loads models, which broke streaming.
    Moving django.setup() in roundwared.rwstreamd fixed the issue.

  - django_chartit is no longer maintained. Moved to django_chartit2
    This may require us to install jQuery and Highcharts manually

  - django.forms.models.save_instance has been removed.
    We are now calling save() instead, but it might need more testing.
    django/django@8656cf
    django/django@b11564

  - Fixed errors in Travis install script (thanks @hburgund)

  - Fixed jsocol/django-adminplus/issues/42

  - Fixed django-admin-bootstrapped dependency
    See cesar57927/django-admin-bootstrapped-1.9-compatible

.travis.yml

@@ -16,6 +16,9 @@ install:
   - sudo su - postgres -c "psql -c \"alter user round password 'round'\""
   - sudo su - postgres -c "psql roundware -c 'create extension postgis'"
   - pip install -r requirements/dev.txt
+  - pip install files/django-admin-bootstrapped-1.9-compatible-master.zip
+# Experiment
+  - sudo apt-get install --reinstall python-setuptools
 before_script:
 # Setup Roundware log file.
   - sudo touch /var/log/roundware
@@ -24,4 +27,4 @@ before_script:
   - sudo mkdir /var/www
   - sudo chmod 777 /var/www
   - export PYTHONPATH=.:/usr/lib/python2.7/dist-packages/
-script: python roundware/manage.py test --settings=roundware.settings.testing
+script: python roundware/manage.py test --settings=roundware.settings.testing

UPGRADING.md

@@ -19,6 +19,54 @@ Done!
 The following instructions describe modifications to the standard upgrade process required due to
 specific changes. Items are listed in reverse chronological order.
 
+### 4/28/16 - Upgrade Django from 1.7 to 1.9
+
+The time has come to upgrade Django and other required apps to their newest versions. If you are
+installing Roundware from scratch, there is no need to take extra steps. However, if your
+installation is based on a commit prior to the Django 1.9 migration, some manual setup is needed.
+
+As part of the upgrade process, [django-guardian](http://django-guardian.readthedocs.io/en/stable/)
+must be updated from 1.2.4 to 1.4.4. While the old versions of guardian used `syncdb` to populate
+the database, newer versions use migrations. The database structures are identical; however, these
+migrations do not check if the tables and relationships are already in place. Therefore, you must
+suppress guardian's initial migration.
+
+1. Make sure that you are in `su - roundware` or `su - vagrant` as appropriate
+2. Pull the relevant post-upgrade roundware-server commit (or newer)
+3. Run `pip install -r ~/roundware-server/requirements/dev.txt`
+   This will download the new requirements, both `common` and `dev`
+   Alternatively, you can `./scripts/runserver.sh` and then `Ctrl+C`
+   It should warn you about unapplied migrations
+4. Navigate to the `roundware` folder in your terminal
+5. Run `python manage.py migrate guardian --fake-initial`
+6. Run `python manage.py migrate` for the other migrations
+
+Note that you might have to also manually uninstall `django-chartit`. `django-chartit2` is meant
+to be a drop-in replacement, but if `django-chartit` is still installed, it will not work. Ensure
+that the `roundware` user can access all new `site-packages`.
+
+You might also need to uninstall `django-admin-bootstrapped` manually. Symptomatically, if the
+Django admin panel has no theme, it's likely that an old version of `d-a-b` was installed globally
+(`/usr/local/lib`) and is now interfering with the new `d-a-b` in the `virtualenv`. Old versions
+required `django_admin_bootstrapped.bootstrap3` to be in `INSTALLED_APPS` to render the theme.
+Otherwise, it would fail silently, and no theme would be rendered. Try running `pip freeze`;
+if `d-a-b` is `v2.0.4`, run `pip uninstall django-admin-bootstrapped`.
+
+Next, you must run the following commands:
+
+ ```
+ rm -f /etc/apache2/sites-available/roundware.conf
+ sed s/USERNAME/roundware/g /var/www/roundware/source/files/etc-apache2-sites-available-roundware > /etc/apache2/sites-available/roundware.conf
+ # If you are using vagrant, replace roundware with vagrant after USERNAME
+ ```
+
+Roundware's `wsgi.py` was moved in this commit, and the Apache conf file must be updated. If you
+get 404 errors after upgrading, chances are you skipped this step.
+
+Lastly, ensure that you are not in a virtual environment, and run `./deploy.sh`
+Check the log to ensure that all of the required apps are located in `/var/www/roundware/lib/`
+It isn't necessary, but it might save you some headache with permissions.
+
 ### 3/7/16 - Convert from MySQL to Postgresql for GIS speaker upgrades
 Related Github issue: https://github.com/roundware/roundware-server/pull/270
 

deploy.sh

@@ -51,6 +51,10 @@ pip install -U pip
 # Install RoundWare requirements
 pip install -r $CODE_PATH/requirements.txt --upgrade
 
+# Loaded in roundware/settings/common.py
+# django-admin-bootstrapped==2.5.7
+pip install ~/roundware-server/files/django-admin-bootstrapped-1.9-compatible-master.zip
+
 # Apply patch to fix M2M field deserializing for Tag relationships, force command to return true.
 # Details: https://code.djangoproject.com/ticket/17946
 # TODO: Remove when fixed in Django core, probably when upgrading to Django 1.8.

files/etc-apache2-sites-available-roundware

@@ -36,7 +36,7 @@
   WSGIDaemonProcess roundware user=USERNAME group=USERNAME umask=002
   WSGIApplicationGroup %{GLOBAL}
   WSGIProcessGroup roundware
-  WSGIScriptAlias / /var/www/roundware/source/files/roundware.wsgi
+  WSGIScriptAlias / /var/www/roundware/source/roundware/wsgi.py
   WSGIPassAuthorization On
 
   # allow CORS for listen map, session map etc

requirements/common.txt

@@ -1,28 +1,26 @@
-Django<1.8
+Django==1.9
 # Creates REST APIs
-djangorestframework==3.2.2
+djangorestframework==3.3.3
 # Used for DRF filtering
-django-filter==0.9
+django-filter==0.13
 # Used in roundware/rw/chart_functions.py
-django-chartit==0.1
+django_chartit2
 # Used in roundware/rw/admin.py, roundware/rw/forms.py, roundware/rw/views, and more.
-django-guardian==1.2.4
+django-guardian==1.4.4
 # Used by roundware/api1/commands.py
-psutil==2.1.3
+psutil==3.4.2
 # Used by roundwared/db.py
 django-cache-utils==0.7.2
 # Used by roundware/rw/fields.py
 django-validated-file==2.0.1
-# Loaded in roundware/settings/common.py
-django-admin-bootstrapped==2.0.4
 # Used in roundware/rw/views.py
 django-braces==1.4.0
 # Loaded in roundware/urls.py
 django-adminplus==0.2.1
 # Used in roundware/rw/forms.py
-django-crispy-forms==1.4.0
+django-crispy-forms==1.6.0
 # Used in roundware/rw/widgets.py
-django-floppyforms==1.2
+django-floppyforms==1.6.1
 # Used in roundware/rw/views.py:
 django-extra-views==0.6.5
 # Used in roundware/rw/fields.py and roundware/rw/widgets.py
@@ -42,6 +40,6 @@ django-cors-headers
 # fiona is a useful tool for processing geographic files (ETL)
 fiona
 # geographic extensions for djangorestframework-gis
-djangorestframework-gis
+djangorestframework-gis==0.10.1
 # leaflet map utilities for django admin
 django-leaflet

requirements/dev.txt

@@ -1,15 +1,15 @@
 -r ./common.txt
 
-django-debug-toolbar
+django-debug-toolbar==1.4
 django-profiler==2.0
 
-#testing.  
-coverage==3.7.1
+#testing.
+coverage==4.0.3
 webtest==2.0.9
-django-webtest==1.7.5
-model_mommy==1.2
-mock==1.0.1
-python-dbusmock==0.8
+django-webtest==1.7.9
+model_mommy==1.2.6
+mock==2.0.0
+python-dbusmock==0.16.3
 
 #interactive interpreter for debugging
-ipython
+ipython==4.0.3

roundware/api2/__init__.py

@@ -1,20 +1,4 @@
 # Roundware Server is released under the GNU Affero General Public License v3.
 # See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
 
-# Contains Roundware DRF REST API V2 signals.
-from __future__ import unicode_literals
-from django.contrib.auth import get_user_model
-from django.db.models.signals import post_save
-from rest_framework.authtoken.models import Token
-import logging
-
-logger = logging.getLogger(__name__)
-
-def create_auth_token(sender, instance=None, created=False, **kwargs):
-    """
-    Create an access Token for every new user
-    """
-    if created:
-        Token.objects.create(user=instance)
-
-post_save.connect(create_auth_token, get_user_model)
+default_app_config = 'api2.apps.RoundwareApi2Config'

roundware/api2/apps.py

@@ -0,0 +1,10 @@
+# Roundware Server is released under the GNU Affero General Public License v3.
+# See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
+
+from django.apps import AppConfig
+
+class RoundwareApi2Config(AppConfig):
+    name = 'api2'
+
+    def ready(self):
+        import roundware.api2.signals

roundware/api2/signals.py

@@ -0,0 +1,20 @@
+# Roundware Server is released under the GNU Affero General Public License v3.
+# See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
+
+# Contains Roundware DRF REST API V2 signals.
+from __future__ import unicode_literals
+from django.contrib.auth import get_user_model
+from django.db.models.signals import post_save
+from rest_framework.authtoken.models import Token
+import logging
+
+logger = logging.getLogger(__name__)
+
+def create_auth_token(sender, instance=None, created=False, **kwargs):
+    """
+    Create an access Token for every new user
+    """
+    if created:
+        Token.objects.create(user=instance)
+
+post_save.connect(create_auth_token, get_user_model)

roundware/rw/filters.py

@@ -3,7 +3,7 @@
 
 from __future__ import unicode_literals
 from django.contrib.admin import DateFieldListFilter, RelatedFieldListFilter
-from django.contrib.admin.util import (get_model_from_relation,
+from django.contrib.admin.utils import (get_model_from_relation,
                                        prepare_lookup_value)
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import smart_unicode

roundware/rw/fixtures/default_auth.json

@@ -818,24 +818,6 @@
   "model": "auth.permission",
   "pk": 91
 },
-{
-  "fields": {
-    "username": "AnonymousUser",
-    "first_name": "",
-    "last_name": "",
-    "is_active": true,
-    "is_superuser": false,
-    "is_staff": false,
-    "last_login": "2015-10-08T13:02:44",
-    "groups": [],
-    "user_permissions": [],
-    "password": "",
-    "email": "",
-    "date_joined": "2015-10-08T13:02:44"
-  },
-  "model": "auth.user",
-  "pk": -1
-},
 {
   "fields": {
     "username": "round",

roundware/rw/fixtures/sample_project.json

@@ -908,22 +908,13 @@
     "model": "rw.speaker",
     "pk": 1
 },
-{
-    "fields": {
-        "client_type": null,
-        "user": -1,
-        "device_id": null
-    },
-    "model": "rw.userprofile",
-    "pk": 1
-},
 {
     "fields": {
         "client_type": null,
         "user": 1,
         "device_id": null
     },
     "model": "rw.userprofile",
-    "pk": 2
+    "pk": 1
 }
 ]

roundware/rw/migrations/0015_remove_null_from_mtm.py

@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.9 on 2016-04-20 15:52
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('rw', '0014__data__speaker_shape_creation'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='asset',
+            name='loc_description',
+            field=models.ManyToManyField(blank=True, to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='asset',
+            name='tags',
+            field=models.ManyToManyField(blank=True, to='rw.Tag'),
+        ),
+        migrations.AlterField(
+            model_name='masterui',
+            name='header_text_loc',
+            field=models.ManyToManyField(blank=True, to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='project',
+            name='demo_stream_message_loc',
+            field=models.ManyToManyField(blank=True, related_name='demo_stream_msg_string', to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='project',
+            name='legal_agreement_loc',
+            field=models.ManyToManyField(blank=True, related_name='legal_agreement_string', to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='project',
+            name='out_of_range_message_loc',
+            field=models.ManyToManyField(blank=True, related_name='out_of_range_msg_string', to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='project',
+            name='sharing_message_loc',
+            field=models.ManyToManyField(blank=True, related_name='sharing_msg_string', to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='tag',
+            name='loc_description',
+            field=models.ManyToManyField(blank=True, related_name='tag_desc', to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='tag',
+            name='loc_msg',
+            field=models.ManyToManyField(blank=True, to='rw.LocalizedString'),
+        ),
+        migrations.AlterField(
+            model_name='tag',
+            name='relationships',
+            field=models.ManyToManyField(blank=True, related_name='_tag_relationships_+', to='rw.Tag'),
+        ),
+    ]