{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":616643637,"defaultBranch":"main","name":"croissant","ownerLogin":"mlcommons","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-03-20T19:44:36.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/67760141?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726820990.0","currentOid":""},"activityList":{"items":[{"before":"f0df159fb40db436ae49c81c9ab2da09ffdd1dfc","after":"759dab3528ccd2f069b9ccdc2090f13c2cb90514","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T13:31:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Fix generation tests.","shortMessageHtmlLink":"Fix generation tests."}},{"before":"089f927befdee7ebd49717621705c2e22c571ce9","after":"f0df159fb40db436ae49c81c9ab2da09ffdd1dfc","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T13:25:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Fix tests and format","shortMessageHtmlLink":"Fix tests and format"}},{"before":"e5ad7ab141c3bd55696926ed1bd97c62e13f602e","after":"089f927befdee7ebd49717621705c2e22c571ce9","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T13:17:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Update metadata for titanic (datasets and cypress)","shortMessageHtmlLink":"Update metadata for titanic (datasets and cypress)"}},{"before":"62bb22ccb1e482161b9163f57c25252e119d34ff","after":"e5ad7ab141c3bd55696926ed1bd97c62e13f602e","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T13:16:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Accept only dictionaries of ids as RecordSets keys","shortMessageHtmlLink":"Accept only dictionaries of ids as RecordSets keys"}},{"before":"d1d85e26f811575e849ce32284f34d88acdda70c","after":"62bb22ccb1e482161b9163f57c25252e119d34ff","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T12:13:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Small typo","shortMessageHtmlLink":"Small typo"}},{"before":"aa066ce9f5fbdea6295cbcc0ecdc3812dc70f5da","after":"d1d85e26f811575e849ce32284f34d88acdda70c","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T12:11:00.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"black formatting","shortMessageHtmlLink":"black formatting"}},{"before":"c749f53e6a4439efeaff75ea8b9388def6d08524","after":"aa066ce9f5fbdea6295cbcc0ecdc3812dc70f5da","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T12:08:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"_value_from_input_types works with dicts of keys","shortMessageHtmlLink":"_value_from_input_types works with dicts of keys"}},{"before":"47a3d86fdaf44ed7554a110d75c7d9da0cf799e9","after":"0b7427b9bdc6ffd46595f400a9f72b3619bfae7e","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-20T09:43:28.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Cache the result of each operation.","shortMessageHtmlLink":"Cache the result of each operation."}},{"before":"de9ea3443b7488b1113d2c8a47ceaa06abe6d598","after":"c749f53e6a4439efeaff75ea8b9388def6d08524","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T09:05:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"ids are a list for mnist","shortMessageHtmlLink":"ids are a list for mnist"}},{"before":null,"after":"de9ea3443b7488b1113d2c8a47ceaa06abe6d598","ref":"refs/heads/list_of_keys","pushedAt":"2024-09-20T08:29:50.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ccl-core","name":null,"path":"/ccl-core","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/91942859?s=80&v=4"},"commit":{"message":"Isolate a `.call()` method in operations. (#736)\n\n- [Simplify graph\r\nprinting.](https://github.com/mlcommons/croissant/pull/736/commits/a3d42557f65add58fb4e2888d1aebe591451a201)\r\n- [Isolate a .call() method in\r\noperations.](https://github.com/mlcommons/croissant/pull/736/commits/36e1d21dc536ceb150aeb6316a5a43aab13d74f8)\r\n\r\nLater, the goal is to have the `.call()` method to populate an internal\r\ncache.\r\n\r\nPseudo-code for next PRs:\r\n\r\n```python\r\nclass Operation:\r\n  _output: Any = dataclass(init=False)\r\n\r\n  def __call__(...):\r\n    if self._output:\r\n      return self._output\r\n    result = self.call(...)\r\n    self._output = result\r\n    return result\r\n```\r\n\r\nThis would allow to retrieve the result of any operation by calling\r\n`operation()` - without argument. This should hopefully simplify the\r\nlogic of making data flow in the graph of operations.\r\n\r\nIn the context of the Beam pipeline, this would allow us to generate\r\nintermediary steps (like intermediary RecordSets for joins for example)\r\nmore easily.\r\n\r\nWhen we don't want to use the cache (e.g., in the case of streamable\r\ndatasets), we can just use operation.call() as we would normally do.","shortMessageHtmlLink":"Isolate a <code>.call()</code> method in operations. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2515604766\" data-permission-text=\"Title is private\" data-url=\"https://github.com/mlcommons/croissant/issues/736\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/mlcommons/croissant/pull/736/hovercard\" href=\"https://github.com/mlcommons/croissant/pull/736\">#736</a>)"}},{"before":"cee60391f69d9678543c885c85028c59bf7f42d5","after":"47a3d86fdaf44ed7554a110d75c7d9da0cf799e9","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-11T12:29:42.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Cache the result of each operation.","shortMessageHtmlLink":"Cache the result of each operation."}},{"before":"525e260afa4a9538264a022b929b177c0bba228d","after":"cee60391f69d9678543c885c85028c59bf7f42d5","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-11T12:21:57.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Cache the result of each operation.","shortMessageHtmlLink":"Cache the result of each operation."}},{"before":"a385d79474d34b9761afe79a29852541cdaefa2f","after":"525e260afa4a9538264a022b929b177c0bba228d","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-11T12:21:25.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Cache the result of each operation.","shortMessageHtmlLink":"Cache the result of each operation."}},{"before":null,"after":"a385d79474d34b9761afe79a29852541cdaefa2f","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-11T12:20:12.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Cache the result of each operation.","shortMessageHtmlLink":"Cache the result of each operation."}},{"before":"36e1d21dc536ceb150aeb6316a5a43aab13d74f8","after":null,"ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-10T08:59:29.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"}},{"before":"94e3f844867bf5eaabbc034edbe10928ef656ea4","after":"de9ea3443b7488b1113d2c8a47ceaa06abe6d598","ref":"refs/heads/main","pushedAt":"2024-09-10T08:59:27.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Isolate a `.call()` method in operations. (#736)\n\n- [Simplify graph\r\nprinting.](https://github.com/mlcommons/croissant/pull/736/commits/a3d42557f65add58fb4e2888d1aebe591451a201)\r\n- [Isolate a .call() method in\r\noperations.](https://github.com/mlcommons/croissant/pull/736/commits/36e1d21dc536ceb150aeb6316a5a43aab13d74f8)\r\n\r\nLater, the goal is to have the `.call()` method to populate an internal\r\ncache.\r\n\r\nPseudo-code for next PRs:\r\n\r\n```python\r\nclass Operation:\r\n  _output: Any = dataclass(init=False)\r\n\r\n  def __call__(...):\r\n    if self._output:\r\n      return self._output\r\n    result = self.call(...)\r\n    self._output = result\r\n    return result\r\n```\r\n\r\nThis would allow to retrieve the result of any operation by calling\r\n`operation()` - without argument. This should hopefully simplify the\r\nlogic of making data flow in the graph of operations.\r\n\r\nIn the context of the Beam pipeline, this would allow us to generate\r\nintermediary steps (like intermediary RecordSets for joins for example)\r\nmore easily.\r\n\r\nWhen we don't want to use the cache (e.g., in the case of streamable\r\ndatasets), we can just use operation.call() as we would normally do.","shortMessageHtmlLink":"Isolate a <code>.call()</code> method in operations. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2515604766\" data-permission-text=\"Title is private\" data-url=\"https://github.com/mlcommons/croissant/issues/736\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/mlcommons/croissant/pull/736/hovercard\" href=\"https://github.com/mlcommons/croissant/pull/736\">#736</a>)"}},{"before":"a3d42557f65add58fb4e2888d1aebe591451a201","after":"36e1d21dc536ceb150aeb6316a5a43aab13d74f8","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-10T07:22:56.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Isolate a `.call()` method in operations.","shortMessageHtmlLink":"Isolate a <code>.call()</code> method in operations."}},{"before":"5fd7a8ea46a2f1d318e5da7deb56a9c06864fe80","after":"a3d42557f65add58fb4e2888d1aebe591451a201","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-10T07:13:01.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Simplify graph printing.","shortMessageHtmlLink":"Simplify graph printing."}},{"before":null,"after":"5fd7a8ea46a2f1d318e5da7deb56a9c06864fe80","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-10T07:12:33.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Simplify graph printing.","shortMessageHtmlLink":"Simplify graph printing."}},{"before":"60ab34583f8b53ea878c4edee88bcad1b2c34523","after":null,"ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-06T10:11:01.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"}},{"before":"5513fac5f8fea6b8e1dc4cd641a9f22ae67a666c","after":"94e3f844867bf5eaabbc034edbe10928ef656ea4","ref":"refs/heads/main","pushedAt":"2024-09-06T10:10:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"New release mlcroissant==1.0.8. (#735)","shortMessageHtmlLink":"New release mlcroissant==1.0.8. (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2509917768\" data-permission-text=\"Title is private\" data-url=\"https://github.com/mlcommons/croissant/issues/735\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/mlcommons/croissant/pull/735/hovercard\" href=\"https://github.com/mlcommons/croissant/pull/735\">#735</a>)"}},{"before":"de727285995f2d77b54fef5de27d568f0b8e3024","after":"60ab34583f8b53ea878c4edee88bcad1b2c34523","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-06T10:10:34.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"New release mlcroissant==1.0.8.","shortMessageHtmlLink":"New release mlcroissant==1.0.8."}},{"before":"9f5605298926e11c7e989f1568f91d2d1792f556","after":"de727285995f2d77b54fef5de27d568f0b8e3024","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-06T09:58:57.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Make the pipeline optional in beam_reader.\n\nWe may not need re-use a pipeline if we want to create a new pipeline\nfrom scratch.","shortMessageHtmlLink":"Make the pipeline optional in beam_reader."}},{"before":"60ab34583f8b53ea878c4edee88bcad1b2c34523","after":"9f5605298926e11c7e989f1568f91d2d1792f556","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-06T09:51:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Make the pipeline optional in beam_reader.\n\nWe may not need re-use a pipeline if we want to create a new pipeline\nfrom scratch.","shortMessageHtmlLink":"Make the pipeline optional in beam_reader."}},{"before":null,"after":"60ab34583f8b53ea878c4edee88bcad1b2c34523","ref":"refs/heads/feature/beam-parallel-operation","pushedAt":"2024-09-06T09:05:28.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"New release mlcroissant==1.0.8.","shortMessageHtmlLink":"New release mlcroissant==1.0.8."}},{"before":"ce10e93eed44b24f6440f6ea34b86c611c718832","after":null,"ref":"refs/heads/feature/fix-beam","pushedAt":"2024-09-05T10:29:06.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"}},{"before":"3a0974ad6af82e13d19f5b599ca4173b89d8153d","after":"5513fac5f8fea6b8e1dc4cd641a9f22ae67a666c","ref":"refs/heads/main","pushedAt":"2024-09-05T10:29:03.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Remove `pipeline` argument from ReadFromCroissant and use `beam.ptransform_fn`. (#734)\n\nDocumentation:\r\nhttps://beam.apache.org/releases/pydoc/2.53.0/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.ptransform_fn","shortMessageHtmlLink":"Remove <code>pipeline</code> argument from ReadFromCroissant and use `beam.ptran…"}},{"before":null,"after":"ce10e93eed44b24f6440f6ea34b86c611c718832","ref":"refs/heads/feature/fix-beam","pushedAt":"2024-09-05T10:15:44.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Remove `pipeline` argument from ReadFromCroissant and use `beam.ptransform_fn`.","shortMessageHtmlLink":"Remove <code>pipeline</code> argument from ReadFromCroissant and use `beam.ptran…"}},{"before":null,"after":"28c2db6cc1a84e51adb365b3630f1a90811f7017","ref":"refs/heads/feature/make-operations-pickable","pushedAt":"2024-09-05T10:12:11.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"},"commit":{"message":"Remove `pipeline` argument from ReadFromCroissant and use `beam.ptransform_fn`.","shortMessageHtmlLink":"Remove <code>pipeline</code> argument from ReadFromCroissant and use `beam.ptran…"}},{"before":"3f198be69c5653a8a3d5d314bff6f85c88e87042","after":null,"ref":"refs/heads/feature/make-operations-pickable","pushedAt":"2024-09-05T08:25:01.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"marcenacp","name":"Pierre Marcenac","path":"/marcenacp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17081356?s=80&v=4"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yMFQxMzozMToyMC4wMDAwMDBazwAAAAS8ACYH","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yMFQxMzozMToyMC4wMDAwMDBazwAAAAS8ACYH","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0wNVQwODoyNTowMS4wMDAwMDBazwAAAAStktes"}},"title":"Activity · mlcommons/croissant"}