Key on EOS: app does not work offline #890
Comments
|
Tested an es preload image: https://images.endlessos.org/builds/12693/ There are findings:
|
erikos
changed the title
|
I think this is going to be fairly straightforward, at least for the case where we've properly preloaded the expected data. For content nodes, I believe that Kolibri will find that everything is present and the scheduled tasks will become no-ops. The issue is channel database imports. Since channel databases are versioned, Kolibri always tries to download them to see if there's a new version available. However, the collections packs already contain the desired version of the databases. We can check that we already have that version of the database available and skip creating a task for it. Skipping content task creation would be harder since we'd have to duplicate all the Kolibri code that determines what content files are needed. It's all pretty hairy as it has to descend the content tree in the channel and apply some heuristics to decide what nodes to include and which files from those nodes are needed. I'm hoping we don't need that. Now, if the preload is missing data that the pack expects, then I don't know how to handle that nicely. We'd need to decide what to do in the case that initial download tasks are failing because of missing network access. Currently it just loops forever trying to restart the tasks over and over. I'm going to consider that handling out of scope for this task. |
|
It's actually news to me that we're scheduling tasks here in the first place. Looking at 53-ek-content-preload, my understanding is the image builder leaves us with a fully populated database and all those content nodes marked Anyway, um, I kind of feel like the explore plugin should not direct the user to the welcome screen if there is content available in the database. That is, if we're creating that whole download state thing for the first time (somewhere inside kolibri_explore_plugin/collectionviews.py), check if there is any content available, and if there is, exit early with "yep, all's good [everything_is_fine.gif]". Separately, I think it would be nice to consider a separate general-purpose "bulk import from kolibri/content/bulk-import" plugin here, which should be easier to test (and to point upstream eventually). I described that in https://github.com/endlessm/endless-key-content-private/issues/102. The bit we would need for the image builder to EOS route is the "code that runs when Kolibri starts up" in this document: https://docs.google.com/document/d/15Yc0Tc4oB9hAO-nFCJ_gNDIuAvsBLKsE5UVhzarNWtc/edit?skip_itp2_check=true&pli=1. One nice thing with that approach is it gives us a way (outside of generating a db.sqlite3 file) for the image builder itself to signal to Kolibri that it has some content to start with, so we aren't doing any guessing and we also know exactly what content pack the user is supposed to be seeing. |
That's the way it did work until #863, but that's too simplistic. We need to apply tags during initial setup and the image builder can't do that since the pack selection needs to be delayed to runtime. If we just checked if there was content when the tags hadn't been applied, then the discovery page would be empty. You could maybe optimize to skip to the tagging if any content exists, but you still need to validate that all the necessary nodes are available or the tagging might fail. So, I think it is appropriate that the downloader is run to ensure that you have all the necessary content. We just have to make it smarter to not actually perform any downloads unless really needed. |
|
Okay, in a separate thread (#898) I'm going to try my hand at the bulk-import stuff that's in my head :) I'm not 100% sure if it'll work out in time so it shouldn't block what you're working on (and those issues you have in mind are 100% worth solving either way). But I'm hoping it'll give us a way to kind of circumvent what we're running into here. |
Kolibri will always make network requests for a channel import task in order to check if the remote version is newer than the local version. However, the starter packs already specify the version we want, so we can check if that's available and skip creating the channel import task. Without that, it's not possible to complete the collection downloader without network access even if all the content is available. Fixes: #890
Kolibri will always make network requests for a channel import task in order to check if the remote version is newer than the local version. However, the starter packs already specify the version we want, so we can check if that's available and skip creating the channel import task. Without that, it's not possible to complete the collection downloader without network access even if all the content is available. Fixes: #890
Kolibri will always make network requests for a channel import task in order to check if the remote version is newer than the local version. However, the starter packs already specify the version we want, so we can check if that's available and skip creating the channel import task. Without that, it's not possible to complete the collection downloader without network access even if all the content is available. Fixes: #890
|
I'm going to make a release so we can consume this in the flatpak and test it out. |
|
This is available to test in 7.11.0 and in the latest flatpak. |
|
@dbnicholson is there a preloaded image available for the test? |
|
I tested the image (eosimpact-eos5.0-amd64-amd64.231027-160050.es_GT.img) today, the issue is still there as:
Flatpak info: eos-diagnostic-231101_144021_UTC+0800.txt I will re-open this issue |
|
In the diagnostics: Hmm, why is it trying to fetch that channel? Shouldn't it already be in the image? Also: Uh, that should only be used in the image builder. At runtime studio.learningequality.org should be used. Something must be wrong in the image builder code. |
This message comes from Kolibri and I think it's a side effect of triggering a channel metadata download without the content server being available. If we need to translate it, it would have to happen upstream. |
|
Oh, wow. If you have any Kolibri option environment variables set when the homedir is first created, they get persisted in the |
That sounds like a reasonable solution. Alternatively, it should be fine to delete Although, indeed, I never noticed that it persists environment variables there. Out of curiousity (and hoping that it would be an easy fix), I checked and it's the same behaviour with Kolibri 0.15. |
|
I tried to debug this directly in kolibri-explore-plugin by mocking a Kolibri homedir with the channels imported the way the image builder does it, but I couldn't reproduce it. I'm flashing the massive 120 GB test image to a USB drive and will try on the real thing. |
|
Ooops... I just realized that https://github.com/endlessm/endless-image-config/pull/1158 has the 3 packs to download as I tried to flash the massive image to a USB drive I had lying around, but it was too small. So today I was going to build a custom image locally and immediately noticed the list of packs was wrong. I'm going to continue with the local build so I can debug it live. |
No, that's wrong as it's supposed to be the English extras pack. |
|
Argh. The image build has the Endless Key flatpak with kolibri-explore-plugin 7.10.0. The fixes from #899 are in 7.11.0. I think after I merged flathub/org.endlessos.Key#20 I must not have waited enough time for it to be published on flathub. This build has some of the image builder changes. Even though there's one more useful change to the flatpak I'd like to have, I'm going to download that image and test it. |
|
QA fail. With the updated image, the channel metadata tasks were being skipped. However, my previous assumption that the content tasks would become essentially no-ops was faulty. In the test suite, a couple Kolibri methods were being mocked, and that hid the fact that Kolibri probes the remote server in this scenario even if it already has all the content objects. So, all tasks really need to be skipped. |
Unfortunately, even if Kolibri already has all the content nodes, it will still probe the remote server for channel metadata. Since that fails when the device is offline, skip creating the tasks if it appears all the content nodes are available. This uses the same `get_import_export_data` helper that Kolibri uses when determining nodes to download. That's an expensive query, but it appears that's the only way to reliably determine if a download is needed or not. Fixes: #890
|
I hope #900 fixes this issue. I've triggered a release of kolibri-explore-plugin, then will get that into the flatpak, then to rebuild the test image. |
|
I'm leaving this open until we get actual verification. |
|
@vanessa-chang can you try the image at https://images.endlessos.org/builds/13181/? |
|
Sure. The download is a bit slow here for me. I will try to update the result soon as I can. |
|
@vanessa-chang was having trouble downloading the image, so I ran the test myself. It works! I turned off the network in my VM and started Endless Key. The downloader progress bar jumps immediately to 90% where it runs the (ridiculously slow) tag application. Then it worked as normal with a populated overview and full library. I think we can go ahead merging and backporting all the things. I'm going to close this kolibri-explore-plugin specific issue and carry on in https://github.com/endlessm/endless-key-content-private/issues/103. |
|
Thanks @dbnicholson for helping the testing. I finally finished the download! I saw the same thing as it ran fast to 90 % and took around 15 seconds to complete the loading. Then everything works well without the internet. |
Follow up from endlessm/eos-image-builder#117
There are some customized images the would need the app content preloaded, so that the user can access it offline.
Content: https://github.com/endlessm/endless-key-content-private/issues/103
AC:
The text was updated successfully, but these errors were encountered: