…As a result an input has multiple windows. So we need to consider that the accumulator can have multiple windows

… with sliding windows

…tions, set testMode to true. Some renaming

…neWithSlidingWindows

…tors, because this method modifies its first input accumulator.

Decrease memory usage by storing only accumulator and timestamp in the combine.merge map

…merged to a window before combining

…oduction mode

…ected results

… using LOOPBACK execution mode

…lidatesRunner" task

… the window and not merge all the timestamps before combine

…ersion-plugin (#10154)

* [BEAM-8729] Gracefully skip irrelevant lines from gradle-version-plugin

* Fixed typo

…10146)

* Add timestamp proto conversion for Python

revert changes

revert changes

* add tests

* lint

* copy work for durations

* lint

…rference with dependency check

…status in beam_Dependency_Check

…nfirm

…10102)

#10148)

This preserves the behavior before PR #10044.

…0135)

…unner to release script

…eck Report

These contain no state or initialization worth managing.
(All caches are shared among all workers.)

Also fixes bug with calling shutdown when tearing down process
bundle executors.

…unner only utilities.

…in FlinkUberJarJobServer

…dule imports.

… on Spark Structured streaming framework

…adge to the github template

…ion-pipeline-group-by-key

[BEAM-6335] Test GBK streaming reading SyntheticSources

…runner tests.

…aluating worker host on Windows until a better solution is available.

…() (#10145)

…ltimaps in shared libraries (#10147)

This now removed the byte[] that was used as the key and exposed the SDKs coder specifically using the structural value for comparison.

Update portable Python to use the iterable state key. Note that this doesn't effect Dataflow since dataflow_runner.py converts all iterable side inputs into multimap right now and no SDK performs key enumeration yet.

Update both Flink and Spark to support iterable API and also key enumeration for multimaps. To minimize the extent of this change, I did the minimal modification for Dataflow. A follow-up PR will do the same for Dataflow and then enable multimap side input key enumeration and iterable lookup within various SDKs.

@kennknowles plus a few other grammar nits and rewrites

… hints.

[BEAM-8795] fix Spark runner build

…a lock to prevent race condition in module imports.

…ervices

…word choice fixes)

…new spark runner validates runner tests.

* [BEAM-8016] Pipeline Graph

1. Removed dependency of pipeline_analyzer, display_manager and
pipeline_graph_renderer from interactive_runner.
2. Replaced the logic of analyzing pipeline with pipeline_instrument
module including how PipelineResult subclass works.
3. Replaced the Pipeline Graph rendering logic with pipeline_graph
module. Implemented the display_graph interface.
4. The cache manager is now held by a static environment object and
shared with all interactive runners. The environment and its cache
manager, however, are intialized lazily by any interactive runner in
current main loop/process/thread.
5. Updated the example ipynotebook.

* lint

* lint

* Watch pipeline defined in unittest for new Interactive Runner logic.

* Merge new Beam Pipeline Graph proposal change that renders PCollections as nodes.

* lint

* disable some pylint warning because it doesn't understand context.

* fix

* fix

* fix

* Remove trailing empty space.

* isort

…ool to aid upgrading dependencies. (#10184)

* [BEAM-7278, BEAM-2530] Add support for using a Java linkage testing tool to aid upgrading dependencies.

For example:
./gradlew -Ppublishing -PjavaLinkageArtifacts=beam-sdks-java-core,beam-sdks-java-io-jdbc :checkJavaLinkage

* Update build.gradle

Co-Authored-By: Tomo Suzuki <[email protected]>

* Update build.gradle

Co-Authored-By: Tomo Suzuki <[email protected]>

…ndle descriptor.

…for the Java SDK harness.

[BEAM-8658] [BEAM-8781] Optionally set jar and artifact staging port …

… list of sources

#9949)

* [BEAM-7948] Add time-based cache threshold support in the Java data service

* [BEAM-7948] Improve the exception handling.

* fix up

* fixup

… adding elements.

[BEAM-8619] Tear down the DoFns upon the control service termination …

…tion to standard

[BEAM-8592] Adjusting ZetaSQL table resolution to standard

…#10195)

* BigQuery Streaming Inserts are always retried by default.

* Fix lint.

* Adding runtime error on errors for always retry.

* Default behaviour is to always retry.

* fix lint