Support MLX in WhisperAX #200
Conversation
…lity (#192) * Make additional initializers, functions, members public, for WKPro * Allows use of default internal functions & member accesses which have increased protections when imported * Initializers were Xcode generated: right click class name -> refactor -> generate memberwise initializers * memberwise initializer defaults to internal, mark as public. * Formatting --------- Co-authored-by: ZachNagengast <[email protected]>
… models (#193) * Add initial mlpackage loading (if .mlmodelc not present) -- Does not modify model loading in OS WK. This is a hook to modify load path URLs. * Always load audio encoder last * Adjust timings to account for decoder<>encoder order swap * Add helper for mlpackage detection --------- Co-authored-by: ZachNagengast <[email protected]>
* Fix start time logic for file loading and resampling * Add test file
As far as I can tell, these stored properties are not meant to be changed. Therefore, change them to be immutable. This change also makes these static properties concurrency-safe.
* Add VoiceActivityDetector base class Add base class to allow different VAD implementations * fix spaces
| ======= | ||
| >>>>>>> main |
There was a problem hiding this comment.
Ah good catch 👍
Package.resolved
Outdated
| "location" : "https://github.com/ml-explore/mlx-swift", | ||
| "location" : "https://github.com/davidkoski/mlx-swift.git", | ||
| "state" : { | ||
| "revision" : "597aaa5f465b4b9a17c8646b751053f84e37925b", | ||
| "version" : "0.16.0" | ||
| "revision" : "3314bc684f0ccab1793be54acddaea16c0501d3c" |
There was a problem hiding this comment.
curious, why this change?
There was a problem hiding this comment.
ok, now I can see why, nvm
There was a problem hiding this comment.
This has been merged into mlx-swift and tagged 0.16.2
| @@ -171,6 +137,74 @@ public struct ModelComputeOptions { | |||
| } | |||
| } | |||
|
|
|||
| public struct ModelInfo: Identifiable, Hashable { | |||
| public let id = UUID() | |||
There was a problem hiding this comment.
curious, why this property public let id = UUID() is needed? can it be uniquely identified by name?
There was a problem hiding this comment.
Needed to make it identifiable, although I was using this for the picker at one point and may be vestigial, will check.
| let keyCache = try? MLX.stacked(keyCacheResult).asMLMultiArray() | ||
| let valueCache = try? MLX.stacked(valueCacheResult).asMLMultiArray() | ||
| let decodingCache = DecodingCache( | ||
| keyCache: keyCache, | ||
| valueCache: valueCache, | ||
| alignmentWeights: nil | ||
| ) | ||
|
|
||
| let logits = try? result.logits?.asMLMultiArray() |
There was a problem hiding this comment.
seems like you've changed it to try ? in couple of places, is it intended?
There was a problem hiding this comment.
Good point, we want this to pass the throw up through the call stack. Reviewing build warnings as well.
|
Hey all, just a heads-up, the example project gets this SPM error:
|
that fork was merged and is now the |
|
Awesome, thanks for the update @davidkoski! |
|
@ZachNagengast - This looks awesome! Apologies - rookie question... It looks like currently the MLX repo only has the base and tiny models. How hard is it for mere mortals to "build" some of the larger models for testing this out with larger models? |
Add support for the WhisperAX example app, as well as various refactors and cleanup.
Note: this is using an unreleased version of MLX, pending merge of ml-explore/mlx-swift#130
There are still some memory issues to address, will look more into this soon.
CoreML:
MLX:
Important note, first time running this will likely throw an error about PrepareMetalShaders, which requires Trust & Enable on this popup when selecting the error.
