Upgrade to Django 1.9

This is a fairly extensive commit, since most of the requirements had
to be upgraded as well. Dependencies have been locked down to specific
versions for consistency and troubleshooting. Notable changes:

  - Moved wsgi.py to a more standard location, updated apache config.
    Now referenced in roundware.settings.WSGI_APPLICATION

  - For roundware.api2, implemented the new Application Configuration
    structure. See http://stackoverflow.com/q/32795227/1943591

  - django-guardian now creates its own AnonymousUser in the database.
    Removed AnonymousUser references from fixtures to avoid conflicts.
    http://django-guardian.readthedocs.io/en/stable/configuration.html

  - django-guardian assign is being depricated for assign_perm
    See django-guardian/django-guardian@1419048

  - ManyToManyField no longer accepts null=True

  - django.contrib.auth.models.User should not be accessed directly.
    Use django.conf.settings.AUTH_USER_MODEL instead

  - Django changed the way it loads models, which broke streaming.
    Moving django.setup() in roundwared.rwstreamd fixed the issue.

  - django_chartit is no longer maintained. Moved to django_chartit2
    This may require us to install jQuery and Highcharts manually

  - django.forms.models.save_instance has been removed.
    We are now calling save() instead, but it might need more testing.
    django/django@8656cf
    django/django@b11564

  - Fixed errors in Travis install script (thanks @hburgund)

  - Fixed jsocol/django-adminplus/issues/42

  - Fixed django-admin-bootstrapped dependency
    See IMAmuseum/django-admin-bootstrapped

  - Fixed queryset filtering in ProjectProtectedThrough classes in admin.py
    Added explicit permission-based queryset filtering to ProjectAdmin

  - Fixed duplicate rows in auth_permission, removed default_auth fixtures.

  - Removed fix-m2m-deserial.patch

This closes roundware#229, fixes roundware#291, and supersedes roundware#282.

.travis.yml

@@ -16,6 +16,7 @@ install:
   - sudo su - postgres -c "psql -c \"alter user round password 'round'\""
   - sudo su - postgres -c "psql roundware -c 'create extension postgis'"
   - pip install -r requirements/dev.txt
+  - sudo apt-get install --reinstall python-setuptools
 before_script:
 # Setup Roundware log file.
   - sudo touch /var/log/roundware
@@ -24,4 +25,4 @@ before_script:
   - sudo mkdir /var/www
   - sudo chmod 777 /var/www
   - export PYTHONPATH=.:/usr/lib/python2.7/dist-packages/
-script: python roundware/manage.py test --settings=roundware.settings.testing
+script: python roundware/manage.py test --settings=roundware.settings.testing

UPGRADING.md

@@ -19,6 +19,59 @@ Done!
 The following instructions describe modifications to the standard upgrade process required due to
 specific changes. Items are listed in reverse chronological order.
 
+### 6/13/16 - Upgrade Django from 1.7 to 1.9
+Related Github issue: https://github.com/roundware/roundware-server/pull/283
+
+The time has come to upgrade Django and other required apps to their newest versions. If you are
+installing Roundware from scratch, there is no need to take extra steps. However, if your
+installation is based on a commit prior to the Django 1.9 migration, some manual setup is needed.
+
+#### Detailed Steps (for production machines, not vagrant)
+
+1. Pull the relevant post-upgrade `roundware-server` commit (or newer)
+2. Remove obsolete `auth_permission` rows (see issue #291):
+
+```
+sudo su - postgres -c 'psql -c "DELETE FROM auth_permission WHERE id IN (SELECT id FROM auth_permission EXCEPT ((SELECT permission_id FROM auth_group_permissions) UNION (SELECT permission_id FROM auth_user_user_permissions)))" roundware'
+```
+
+3. Initial deploy: `sudo ./deploy.sh` (this will error on `django-guardian` but don't worry,
+   we're about to fix that)
+4. Run migrations
+
+ ```
+ sudo su - roundware -c "/var/www/roundware/source/roundware/manage.py migrate guardian --fake-initial"
+ sudo su - roundware -c "/var/www/roundware/source/roundware/manage.py migrate"
+ ```
+5. Replace apache config (note this will over-write any customizations you may have made)
+
+ ```
+ sudo rm -f /etc/apache2/sites-available/roundware.conf
+ sudo su - -c "sed s/USERNAME/roundware/g /var/www/roundware/source/files/etc-apache2-sites-available-roundware > /etc/apache2/sites-available/roundware.conf"
+ ```
+6. Run `sudo ./deploy.sh`
+
+#### Notes / Troubleshooting
+
+* As part of the upgrade process, [django-guardian](http://django-guardian.readthedocs.io/en/stable/)
+must be updated from 1.2.4 to 1.4.4. While the old versions of guardian used `syncdb` to populate
+the database, newer versions use migrations. The database structures are identical; however, these
+migrations do not check if the tables and relationships are already in place. Therefore, you must
+suppress guardian's initial migration.
+* You might have to manually uninstall `django-chartit`. `django-chartit2` is meant
+to be a drop-in replacement, but if `django-chartit` is still installed, it will not work. Ensure
+that the `roundware` user can access all new `site-packages`.
+* You might also need to uninstall `django-admin-bootstrapped` manually. Symptomatically, if the
+Django admin panel has no theme, it's likely that an old version of `django-admin-bootstrapped` was installed globally
+(`/usr/local/lib`) and is now interfering with the new `django-admin-bootstrapped` in the `virtualenv`. Old versions
+required `django_admin_bootstrapped.bootstrap3` to be in `INSTALLED_APPS` to render the theme.
+Otherwise, it would fail silently, and no theme would be rendered. Try running `pip freeze`;
+if `django-admin-bootstrapped` is `v2.0.4`, run `pip uninstall django-admin-bootstrapped`.
+* Roundware's `wsgi.py` was moved in this commit, and the Apache conf file must be updated. If you
+get 404 errors after upgrading, chances are you skipped this step.
+* Check the log to ensure that all of the required apps are located in `/var/www/roundware/lib/`.
+It isn't necessary, but it might save you some headache with permissions.
+
 ### 3/7/16 - Convert from MySQL to Postgresql for GIS speaker upgrades
 Related Github issue: https://github.com/roundware/roundware-server/pull/270
 

deploy.sh

@@ -48,14 +48,9 @@ export PYTHONPATH=$CODE_PATH
 # Install upgrade pip
 pip install -U pip
 
-# Install RoundWare requirements
+# Install Roundware requirements
 pip install -r $CODE_PATH/requirements.txt --upgrade
 
-# Apply patch to fix M2M field deserializing for Tag relationships, force command to return true.
-# Details: https://code.djangoproject.com/ticket/17946
-# TODO: Remove when fixed in Django core, probably when upgrading to Django 1.8.
-patch -N $WWW_PATH/lib/python2.7/site-packages/django/core/serializers/python.py < $CODE_PATH/files/fix-m2m-deserial.patch || true
-
 # Set $USERNAME to own WWW_PATH files
 chown $USERNAME:$USERNAME -R $WWW_PATH
 

files/etc-apache2-sites-available-roundware

@@ -36,7 +36,7 @@
   WSGIDaemonProcess roundware user=USERNAME group=USERNAME umask=002
   WSGIApplicationGroup %{GLOBAL}
   WSGIProcessGroup roundware
-  WSGIScriptAlias / /var/www/roundware/source/files/roundware.wsgi
+  WSGIScriptAlias / /var/www/roundware/source/roundware/wsgi.py
   WSGIPassAuthorization On
 
   # allow CORS for listen map, session map etc

requirements/common.txt

@@ -1,28 +1,26 @@
-Django<1.8
+Django==1.9
 # Creates REST APIs
-djangorestframework==3.2.2
+djangorestframework==3.3.3
 # Used for DRF filtering
-django-filter==0.9
+django-filter==0.13
 # Used in roundware/rw/chart_functions.py
-django-chartit==0.1
+django_chartit2
 # Used in roundware/rw/admin.py, roundware/rw/forms.py, roundware/rw/views, and more.
-django-guardian==1.2.4
+django-guardian==1.4.4
 # Used by roundware/api1/commands.py
-psutil==2.1.3
+psutil==3.4.2
 # Used by roundwared/db.py
 django-cache-utils==0.7.2
 # Used by roundware/rw/fields.py
 django-validated-file==2.0.1
-# Loaded in roundware/settings/common.py
-django-admin-bootstrapped==2.0.4
 # Used in roundware/rw/views.py
 django-braces==1.4.0
 # Loaded in roundware/urls.py
 django-adminplus==0.2.1
 # Used in roundware/rw/forms.py
-django-crispy-forms==1.4.0
+django-crispy-forms==1.6.0
 # Used in roundware/rw/widgets.py
-django-floppyforms==1.2
+django-floppyforms==1.6.1
 # Used in roundware/rw/views.py:
 django-extra-views==0.6.5
 # Used in roundware/rw/fields.py and roundware/rw/widgets.py
@@ -42,6 +40,8 @@ django-cors-headers
 # fiona is a useful tool for processing geographic files (ETL)
 fiona
 # geographic extensions for djangorestframework-gis
-djangorestframework-gis
+djangorestframework-gis==0.10.1
 # leaflet map utilities for django admin
-django-leaflet
+django-leaflet==0.18.0
+# Bootstrap admin theme for Django 1.9
+https://github.com/roundware/django-admin-bootstrapped/zipball/3.0.0

requirements/dev.txt

@@ -1,15 +1,15 @@
 -r ./common.txt
 
-django-debug-toolbar
+django-debug-toolbar==1.4
 django-profiler==2.0
 
-#testing.  
-coverage==3.7.1
+#testing.
+coverage==4.0.3
 webtest==2.0.9
-django-webtest==1.7.5
-model_mommy==1.2
-mock==1.0.1
-python-dbusmock==0.8
+django-webtest==1.7.9
+model_mommy==1.2.6
+mock==2.0.0
+python-dbusmock==0.16.3
 
 #interactive interpreter for debugging
-ipython
+ipython==4.0.3

roundware/api2/__init__.py

@@ -1,20 +1,4 @@
 # Roundware Server is released under the GNU Affero General Public License v3.
 # See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
 
-# Contains Roundware DRF REST API V2 signals.
-from __future__ import unicode_literals
-from django.contrib.auth import get_user_model
-from django.db.models.signals import post_save
-from rest_framework.authtoken.models import Token
-import logging
-
-logger = logging.getLogger(__name__)
-
-def create_auth_token(sender, instance=None, created=False, **kwargs):
-    """
-    Create an access Token for every new user
-    """
-    if created:
-        Token.objects.create(user=instance)
-
-post_save.connect(create_auth_token, get_user_model)
+default_app_config = 'api2.apps.RoundwareApi2Config'

roundware/api2/apps.py

@@ -0,0 +1,10 @@
+# Roundware Server is released under the GNU Affero General Public License v3.
+# See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
+
+from django.apps import AppConfig
+
+class RoundwareApi2Config(AppConfig):
+    name = 'api2'
+
+    def ready(self):
+        import roundware.api2.signals

roundware/api2/signals.py

@@ -0,0 +1,20 @@
+# Roundware Server is released under the GNU Affero General Public License v3.
+# See COPYRIGHT.txt, AUTHORS.txt, and LICENSE.txt in the project root directory.
+
+# Contains Roundware DRF REST API V2 signals.
+from __future__ import unicode_literals
+from django.contrib.auth import get_user_model
+from django.db.models.signals import post_save
+from rest_framework.authtoken.models import Token
+import logging
+
+logger = logging.getLogger(__name__)
+
+def create_auth_token(sender, instance=None, created=False, **kwargs):
+    """
+    Create an access Token for every new user
+    """
+    if created:
+        Token.objects.create(user=instance)
+
+post_save.connect(create_auth_token, get_user_model)

roundware/rw/admin.py

@@ -62,23 +62,30 @@ def get_queryset(self, request):
             return qset
 
         accessible_projects = get_objects_for_user(
-            request.user, 'rw.access_project')
+            request.user, 'rw.access_project', Project)
         authorized_objects = model_class.objects.filter(
             project__in=accessible_projects)
         return qset.filter(**{field_name + "__in": authorized_objects})
+
     return get_queryset
 
 
 class ProjectProtectedThroughAssetModelAdmin(admin.ModelAdmin):
-    queryset = project_restricted_queryset_through(Asset, 'asset')
+
+    def get_queryset(self, request):
+        return project_restricted_queryset_through(Asset, 'asset')(self, request)
 
 
 class ProjectProtectedThroughSessionModelAdmin(admin.ModelAdmin):
-    queryset = project_restricted_queryset_through(Session, 'session')
+
+    def get_queryset(self, request):
+        return project_restricted_queryset_through(Session, 'session')(self, request)
 
 
 class ProjectProtectedThroughUIModelAdmin(admin.ModelAdmin):
-    queryset = project_restricted_queryset_through(MasterUI, 'master_ui')
+
+    def get_queryset(self, request):
+        return project_restricted_queryset_through(MasterUI, 'master_ui')(self, request)
 
 
 class ProjectProtectedModelAdmin(admin.ModelAdmin):
@@ -89,7 +96,22 @@ def get_queryset(self, request):
         if request.user.is_superuser:
             return qset
 
-        return qset.filter(project__in=get_objects_for_user(request.user, 'rw.access_project'))
+        # TODO: Consider using an M2M field instead, if more complex permissions not needed:
+        # models.py: user = models.ManyToManyField(settings.AUTH_USER_MODEL, blank=True)
+        # admin.py: accessible_projects = Project.objects.filter(user=request.user)
+
+        accessible_projects = get_objects_for_user(request.user, 'rw.access_project', Project)
+
+        return qset.filter(project__in=accessible_projects)
+
+class ProjectModelAdmin(GuardedModelAdmin):
+
+     def get_queryset(self, request):
+
+        if request.user.is_superuser:
+            return super(admin.ModelAdmin, self).get_queryset(request)
+
+        return get_objects_for_user(request.user, 'rw.access_project', Project)
 
 
 def copy_asset(modeladmin, request, queryset):
@@ -267,7 +289,7 @@ class VoteAdmin(ProjectProtectedThroughAssetModelAdmin):
     ordering = ['id']
 
 
-class ProjectAdmin(GuardedModelAdmin):
+class ProjectAdmin(ProjectModelAdmin):
     list_display = ('id', 'name', 'latitude', 'longitude',
                     'max_recording_length', 'recording_radius')
     ordering = ['id']
@@ -295,7 +317,6 @@ class ProjectAdmin(GuardedModelAdmin):
         }),
     )
 
-
 class SessionAdmin(ProjectProtectedModelAdmin):
     list_display = ('id', 'project', 'starttime', 'device_id', 'language')
     list_filter = ('project', 'language', 'starttime')

roundware/rw/filters.py

@@ -3,7 +3,7 @@
 
 from __future__ import unicode_literals
 from django.contrib.admin import DateFieldListFilter, RelatedFieldListFilter
-from django.contrib.admin.util import (get_model_from_relation,
+from django.contrib.admin.utils import (get_model_from_relation,
                                        prepare_lookup_value)
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import smart_unicode