Update of Habana SynapseAI notebook to 1.15.1 #533
Conversation
|
@harshad16 PTAL |
|
/retest |
|
@Xaenalt , i m not sure if we would like to maintain python 3.10 to be honest |
|
/hold |
|
Yeah, thankfully the same versions of everything exist in 3.10, but yeah let's have a longer discussion about it |
|
According to Intel, this would only be necessary until the 1.17 release which will be end of July-ish |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Red Hat ships Python 3.11, can you wait and use that? It is supported until its retirement date in May 2026 in rhel 8 as well as rhel 9. If you base the image on ub9/rhel9 and use Python 3.12 available there, that has retirement date in Apr 2027. https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle#rhel8_application_streams |
This required quite a few changes, since the SynapseAI stack requires Python 3.11 for 1.15.1. Uses a similar workflow to Anaconda with a custom base image, this however should be easy to update in the future just by swapping out the FROM in the base image and verifying the python versions going forward. Signed-off-by: Sean Pryor <[email protected]>
In loading Torch (and other modules), it prints out the following Warning: please export TSAN_OPTIONS='ignore_noninstrumented_modules=1' to avoid false positive reports from the OpenMP runtime! I added this to an ENV section in the image, since it seems worth squashing these false positive reports, I left it as an extra commit though in case others want to leave that to the user
|
If we're good to use Python 3.11 that'll be excellent, and will be supported in the 1.17 release I was just doing a few small fixes to the notebook PR, since it's a much easier to use template for the future |
|
If possible, I'd like to revisit the timeline until 1.17 (EO July timeframe) at which point we can retire the 1.15/1.16 series which are stuck on Python 3.10. Currently (pre 1.17) the notebook SynapseAI version has to match what's in the operator |
|
@Xaenalt JFYI, I updated the description of the https://issues.redhat.com/browse/RHOAIENG-5404 which is tracking addition of the Gaudi v1.17 into RHOAI as v1.17 seems to be out now. In the meantime, there is a plan to introduce a Python 3.11 via #659 (just the image definitions but won't be used in the actual RHOAI/ODH builds yet - that will be a next step). |
| PROMPT_COMMAND=". ${APP_ROOT}/bin/activate" | ||
|
|
||
| # Squash a warning from the Habana runtime | ||
| ENV TSAN_OPTIONS='ignore_noninstrumented_modules=1' |
There was a problem hiding this comment.
are we really compiling something with thread sanitizer enabled? this gives you 3 to 10x more memory consumption and about the same amount of performance degradation; tsan is intended for debug builds only
| tar -xzvf /tmp/openshift-client-linux.tar.gz oc && \ | ||
| rm -f /tmp/openshift-client-linux.tar.gz | ||
|
|
||
| # Fix permissions to support pip in Openshift environments |
There was a problem hiding this comment.
this should be done in the same RUN as the micropipenv install that created these files
|
@Xaenalt: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I think that this work will be superseded by the #695. |
|
Yep, superseded by #695 |
Description
Adds the SynapseAI 1.15.1 notebook, which is needed for RHOAI 2.10 compatibility
This required quite a few changes, since the SynapseAI stack requires Python 3.11 for 1.15.1. Uses a similar workflow to Anaconda with a custom base image, this however should be easy to update in the future just by swapping out the FROM in the base image and verifying the python versions going forward.
How Has This Been Tested?
Builds successfully, could use some functional testing
Merge criteria: