Skip to content

lessons

Jump to bottom
Miguel Gamboa edited this page May 22, 2023 · 28 revisions

Lessons:

Bibliography - The Well-Grounded Java Developer, Second Edition:

  • 1.1 - The language and the platform
  • 1.2 - The new Java release model
  • 9,6 - Java interoperability
  • 4.1 - Class loading and class objects
  • 4.2 - Class loaders
  • 4.3 - Examining class files
  • 4.5 - Reflection
  • 4.5 - Bytecode
  • 7.1 - Performance terminology: Some basic definitions
  • 7.2 - A pragmatic approach to performance analysis
  • 7.3 - What went wrong? Why do we have to care?
  • 7.4 - Why is Java performance tuning hard?
  • 7.5 - Garbage collection
  • 15.3.7 - Sequences
  • 16.3 - Under the hood with Kotlin coroutines

Lesson 01 - Introduction and Modern VMs, e.g. JVM, Node

  • Bibliography and Lecturing methodology: github and slack.
  • Tools: javac, javap, kotlinc, JDK 17 and gradle.
  • Program outline in 3 parts:
    1. Java Type System and Reflection;
    2. Metaprogramming and Performance;
    3. Iterators versus Sequences (yield). Generics and Reified type parameters.
  • Project in 3 parts according to program outline.
  • Managed Runtime or Execution Environment.
  • Informally virtual machine (VM) or runtime
  • Historic evolution since Java to nowadays
  • Examples of languages targeting JVM: Java, kotlin, Scala, Clojure.
  • Examples of languages targeting Node: JavaScript, TypeScript, Kotlin.
  • Java syntax similar to C and C++.
  • Distinguish between Programming Language <versus> VM

Lesson 02 - Java Type System, Components and Metadata

  • Type System - Set of rules and principles that specify how types are defined and behave.
  • Module .classfor each class definition
  • CLASSPATH
    • e.g. -cp . - local folder
    • e.g. -cp .:JetBrains/IntelliJIdea2022.2/plugins/Kotlin/lib/*
    • (for windows use ; rather than :)
  • Class have Members
  • Members may be: Fields, Constructors or Methods.
  • Constructor is a function with name "<init>" returning void.
  • Using javap -p Student.class to inspect metadata
  • Using javap -c -p AppKt.class to inspect metadata and bytecode definition of data class Point
  • Analyzing kotin properties in Java.
  • There are NO properties in Java Type System.
  • A Kotlin property may generate:
    • Backing field -- accessed with getfield bytecode.
    • Getter, i.e. function get... -- called with invoke... bytecode.
    • Setter, i.e. function set... (if defined with var).
  • Kotlin val <=> Java final.
  • FIELDS => MEM
  • bytecode invokevirtual to call e.g. getNr() and print()
  • Component - Reusable software unit, with:
    • IR - code in intermediate representation (e.g. bytecode, IL, other)
    • Metadata - auto description
    • Ready to use => does not require static compilation.
    • API => conforms to a public interface.
    • Indivisible => 1 module (file)
  • Software development by Components:
    • Reduce Complexity;
    • Promote Reuse.
  • Developer roles:
    • Provider - provides a component with a well-known API (e.g. Point)
    • Client - uses the component from a provider to build an Application, or another component (e.g. App).
  • Managed Runtime:
    • Software components = Metadata + IR (intermediate representation) (e.g. bytecode)
    • Interoperability supported at bytecode and metadata level e.g. Java <> Kotlin
    • Portability - Compile once and run everywhere with VM, e.g. JRE.
    • Jitter - Just-in-time compiler
    • Safety - NO invalid memory accesses
    • GC - Garbage Collector.
    • ClassLoader and CLASSPATH - dynamic and lazy load.

Lesson 03 - Kotlin to JVM

  • Homework 01 resolution
  • Java Type descriptors
  • Java access control
  • Immutability through final fields equivalent to kotlin val
  • Virtual and non-virtual methods (Kotlin open versus Java final )
  • Optional @Override annotation in Java equivalent to kotlin keyword override
  • Java static members
  • Translating kotlin companion objects to Java
  • Translating kotlin extensions to Java

Lesson 04 - Managed versus Unmanaged

  • Unmanaged <versus> Managed
  • Static <versus> Dynamic link
  • Building unmanaged components with static link:
    • NOT a unit, but two parts instead: header + obj
    • Need of a header file that describes the component content
    • Different builds for different architectures
    • Structural modifications (new header) => compilation + link
    • Behavioral modifications => link
  • Demo with an unmanaged App using a Point.
    • Static compilation (gcc -c) and static link (gcc)
  • Demo with a managed App using a Point.
    • Jitter (just-in-time compiler) - compiles IR to native code (e.g. x86, amd64, ppc) at runtime
    • Dynamic link - Point.class on App compilation + link at runtime.
    • Lazy Load - Point.class is loaded only when needed

Lesson 05 - Programming with Metadata

  • Homework 02 resolution
  • Translating kotlin companion objects to Java
  • Java static nested classes e.g. class Person { static class Companion { .. }} => Person$Companion
  • Java static constructor
  • Singleton design pattern
  • Private instance constructor to avoid new objects.
  • Translating kotlin extensions to Java.
  • Reflection object oriented API for metadata
  • KClass is representing a type in Kotlin, whereas Class is representing a type in JVM.
  • KClass --- .java ---> Class
  • Class ---- .kotlin ---> KClass
  • TypeName.class - returns the Class corresponding to the type TypeName.
  • refObject.getClass() - returns the Class of the object referenced by refObject.
  • Java Reflection API (java.lang.reflect): Class, Member, Field, Executable, Constructor, Method and Parameter.
  • Inspecting functions trough Reflection API
  • Method => Class<?> getReturnType(); Parameter[] getParameters();
  • Parameter => Class<?> getType() - the declared type for the parameter represented by this Parameter.
  • Field => Class<?> getType() - the declared type for the field represented by this Field.
  • NOTICE do not mislead getType() with getClass()
  • Invoking methods and creating instances:
  • Method => Object invoke(Object obj, Object... args)
    • Invokes the underlying method represented by this Method object, on the specified object obj with the specified parameters args.
  • Check if a method is static - Modifier.isStatic(m.getModifiers())
  • Constructor => T newInstance(Object... initargs)
    • Uses the constructor represented by this Constructor object to create and initialize a new instance of the constructor's declaring class, with the specified parameters initargs.
  • ClassLoader - class-loading capability exposed to the user programmer.
    • BootstrapClassLoader - used to get the absolute basic system loaded—essentially java.base.
    • AppClassLoader - loads the application classes in CLASSPATH.
    • UrlClassLoader - loads classes and resources from a search path of URLs.
  • Example
static Stream<Class<?>>listClassesInClasspath() throws IOException {
    ClassLoader cl = ClassLoader.getSystemClassLoader();     // An AppClassLoader instance
    return list(cl.getResources("")).stream()
            .filter(url -> url.getProtocol().equals("file")) // Exclude jar files
            .map(url -> new File(toURI(url)))
            .flatMap(f -> f.isDirectory() ? stream(f.listFiles()) : Stream.of(f))
            .filter(f -> f.getName().contains(".class"))
            .map(f -> loadClass(cl, qualifiedName(f.getName())));
}

Lesson 06 - Programming with Metadata

  • Homework 03 resolution
  • Distinguishing between streams of java.io and java.util.stream.
  • java.io streams are related with implementations of InputStream and OutputStream.
  • java.util.stream provides abstractions on sequences and its operations like filter, map, reduce, etc.
  • Type check, e.g. if m is compatible with Method:
    • m instanceof Method => true if m is an instance of the class Method or any inherited class.
    • m.getClass() == Method => true if m has exactly the type Method.
  • get<member> versus getDeclared<member> to get inaccessible members (e.g. private).
  • setAccessible(boolean) => to invoke inaccessible methods or constructors (e.g. private)

Lesson 07 - Logger via reflection

  • Distinguishing the role of App domain ---> Logger utility.
  • Unit tests with sample domain Student and Point
  • Simple Logger ---> Printer
  • Decouple Logger output into a Printer interface.
  • Implement PrinterConsole and PrinterBuffer.
  • Make PrinterConsole singleton.
  • logFields(Object target)
  • logProperties(Object target) - for parameterless methods with get prefix and non void return type.

Lesson 08 - Logger and interface Getter and Annotations

  • Maintain getters on Map<Class<?>, List<Getter>>
  • interface Getter { void getAndPrint(target: Object) }
  • Distinct implementations: loadFieldGetters(), loadPropertyGetters(), ...
  • Annotations
  • isAnnotationPresent(Class<? extends Annotation> annotationClass)
  • getAnnotation(Class<T> annotationClass)
  • Retention
  • @NonLog and AltName(name="<alternate name>"

Lesson 11 - Logger and Annotations with Class parameter

  • Customizing the value printed by Logger.
  • How to pass a function to annotations?
  • Use Class as annotation parameter.

Lesson 14 - Reflection Exercises

  1. Develop the utility Java function Comparator<Object> comparerOrder(Class<?> klass) that returns an instance of Comparator that is able to compare objects of the type represented by klass, according to the properties which are both: Comparable and annotated with Comparison. Notice that you should compare respecting the order specified in annotation. Example:
class Student (
  @Comparison(2) val nr:Int,
  val name: String,
  @Comparison(1) val nationality: String,
)
 
val s1 = Student(12000, "Ana", "pt")
val s2 = Student(14000, "Ana", "pt")
val s3 = Student(11000, "Ana", "en")
val cmp = comparerOrder(Student::class)
assertTrue { cmp.compare(s1, s2) < 0 } // same nationality and 12000 is < 14000
assertTrue { cmp.compare(s2, s3) > 0 } // “pt” is > “en”
  1. Develop the utility Java function Comparator<Object> comparer(Class<?> klass) that returns a new intance of Comparator and it is able to compare instances of type represented by klass, according to the properties which are: Comparable OR annotated with a Comparison that specifies a Comparator for that property, according to the following example:
class Person(
  val id: Int,
  val name: String,
  @Comparison(cmp = AddressByRoad::class) val address: Address,
  @Comparison(cmp = AccountByBalance::class) val account: Account) {
}

class AccountByBalance : Comparator<Account>{
  override fun compare(o1: Account, o2: Account): Int {
    return o1.balance.compareTo(o2.balance);
  }
}

class AddressByRoad : Comparator<Address> {
  override fun compare(o1: Address, o2: Address): Int {
    return o1.road.compareTo(o2.road)
  }
}
 
val p1 = Person(11000, "Ana", Address("Rua Amarela", 24), Account("FD3R", 9900))
val p2 = Person(11000, "Ana", Address("Rua Rosa", 24), Account("8YH5", 9900))
val p3 = Person(11000, "Ana", Address("Rua Rosa", 24), Account("JK2E", 100))
val p4 = Person(11000, "Ana", Address("Rua Rosa", 97), Account("BFR5", 100))
val p5 = Person(17000, "Ana", Address("Rua Rosa", 97), Account("BFR5", 100))
val cmp = Comparer<Person>(Person::class)

assertTrue { cmp.compare(p1, p2) < 0 } // Rua Amarela is < Rua Rosa
assertTrue { cmp.compare(p2, p3) > 0 } // 9900 is > 100
assertEquals(0, cmp.compare(p3, p4))   // All properties are equal
assertTrue { cmp.compare(p4, p5) < 0 } // 11000 is < 17000

Lesson 14 - Bytecode

  • Reference Types are instantiated in bytecode with:
    • new - Allocates storage on Heap, initializes space and the object's header, and returns the reference to newbie object.
    • invokespecial - Call to class <init> method (corresponding to constructor).
  • Instantiating a refence type, e.g. Student(765134, "Ze Manel") may produce in bytecode:
new           #8   // class Student
dup                // duplicates the value on top of the stack
...                // One load (push) for each parameter of <init> (constructor)
invokespecial #14  // Method Student."<init>"
  • invokestatic - no thisrequired
  • invokespecial - static dispatch (i.e. non polymorphic)
  • invokevirtual - dynamic dispatch (i.e. polymorphic)
  • invokeinterface - dynamic dispatch (i.e. polymorphic) for interface methods
  • Constant Pool
  • 16-bit index into the constant pool
  • Load and store opcodes
  • Shortcut opcode forms
  • Arithmetic
  • Execution Flow

Lesson 15 - Benchmarking

  • Benchmark - assess the relative performance
  • Benchmark != Unit Tests
  • A naif approach - direct measurement, e.g.
    • measureTimeMillis { logger.log(Student(...)) }.also { println("logger.log() took $it millis"); }
  • Some Problems:
    1. Mixing domain instantiation (i.e. Student) with operation execution logger.log().
    2. First execution includes Jitter overhead and misses optimizations.
    3. Milliseconds could not be accurate enough.
    4. IO may be orders of magnitude slower than log operation itself
    5. IDE (e.g. InteliJ) may induce other overheads.
    6. Absolute results in milliseconds may vary on different hardware environments.
    7. System.currentTimeMillis() includes a System call with implicit overhead.
    8. Garbage Collector may degrade performance
  • Minimize side effects:
    1. Remove domain instantiation from operation measurement
    2. Include warm-up => Optimizations may improve performance
    3. Measure the total execution of several iterations rather than several measurements of single executions.
    4. Avoid IO => Mocking IO
    5. Avoid extra tools such as IDE (e.g. InteliJ), gradle, or other => run directly on VM (e.g. java)
    6. Baseline => How much can we improve performance?
    7. same as 3.
    8. GC => Run several iterations and discard most divergent results.
  • JMH - Java Microbenchmark Harness.
  • Benchmark tests annotated with @Benchmark.
  • JMH Gradle Plugin
  • gradlew jmhJar
  • java -jar <path to JAR> -f 1 -wi 4 -i 4 -w 2 -r 2 -tu ms:
    • -f - forks
    • -wi - warm-up iterations
    • -i - iterations
    • -w - each warm-up iteration duration
    • -r - each iteration duration
    • -tu - time unit

Lesson 16 - Meta-programming and Cojen Maker

  • Introduction to Metaprograming and dynamic code generation;
  • Cojen Maker API: ClassMaker, MethodMaker, FieldMaker
  • finish(): Class, finishTo(OutputStream)
  • Auxiliary function printBytecodes(bytes: ByteArray) of autorouter test project.

Lesson 17 - LoggerDynamic with Cojen Maker

  • Logger refactoring with a base type AbstractLogger extended by every logger approach: baseline, reflect or dynamic.
  • Implementing a dynamic builder for a logger using Cojen Maker:
    • ClassMaker buildLoggerDynamicForProperties(Class<?> domain)

Lesson 18 - Benchmarking

  • Saving from the dead-code elimination.
  • JMH Blackhole - "consumes" the values, conceiving no information to JIT whether the value is actually used afterwards.
  • JMH API: @Setup, @Param, @State
  • Comparing performance between: baseline, reflect or dynamic.

Lesson 22 - Boxing Unboxing

  • Casting between reference types:
    • upcasting in bytecodes it is only copying references, e.g. aload_0 and astore_1
    • downcasting includes checkcast bytecode
  • checkcast checks for compatibility (i.e. same type or subtype).
  • checkcast throws ClassCastException on unsuccess, otherwise leaves the object's reference on top of the stack.
  • Primitive <-> Reference: boxing or unboxing:
    • There is no specific JVM bytecode for these conversions.
    • Supported in JVM through auxiliary functions (i.e. valueOf() and <type>Value()) of Wrapper classes.
  • Primitive Type -> Reference: boxing through <Wrapper Type>.valueOf(primitive)
    • e.g. fun boxing(nr: Int) : Any { return nr }
  • Reference Type -> Primitive: unboxing through <Wrapper>.<primitive>Value()
    • e.g. fun unboxing(nr: Int?) : Int { return nr ?: throw Exception("Null not supported for primitive Int!") }
Types Java Kotlin
Value int, long, double, ... Int, Long, Double, ...
Reference Integer, Long, Double, ... Int?, Long?, Double?, ...

Lesson 25 - Garbage Collector

  • Local values in Stack <versus> Objects in Heap.
  • Local variables managed on stack <versus> Objects managed on Heap by Garbage Collector
    • Stack frame on function's execution => cleaned on function completion, including local variables.
    • Unreachable objects are candidates for Garbage Collector
  • Object's layout and header (containing type information):
    • header is useful for operations such as obj::class (<=> obj.getClass() in Java)
  • Garbage Collection - "process of looking at heap memory, identifying which objects are in use and which are not, and deleting the unused objects."
  • Unused object, or unreferenced object.
  • In use object, or a referenced object:
    • "some part of your program still maintains a pointer to that object."
    • Referenced by a root or belongs to a graph with a root reference.
    • root reference:
      • Local variables - stored in the stack of a thread.
      • Static variables - belong to the class type stored in permanent generation.
  • GC basic process: 1. Marking and 2. Deletion.
  • Deletion approaches:
    • Normal Deletion => holds a list to free spaces. !! memory allocation slower !!
    • Deletion with Compacting => move referenced objects together ++ makes new memory allocation faster !!! longer garbage collection time !!!
  • Generational Garbage Collection:
    • Most objects are short lived => GC is more effective in younger generations.
    • Enhance the performance
  • JVM Generations:
    • Young Generation is where all new objects are allocated and aged.
    • Old Generation is used to store long surviving objects.
    • Permanent generation contains metadata, classes and methods.
  • "Stop the World" events:
    • minor garbage collection - collects the young generation.
    • major garbage collection - collects the old generation.
  • Young Generation process: Eden, Survivor Space 0 and Survivor Space 1.

Lesson 26 - Sequences

  • Distinguishing between:
    • General-purpose language, e.g. Kotlin, Java C#,
    • Domain-specific languages, e.g. SQL, Jetpack Compose, Linq,
  • E.g. SQL for relational data, e.g. SELECT COUNT (DISTINCT ...) FROM ... WHERE ...
  • E.g. Jetpack Compose application { Window { MaterialTheme { ... } Column {... } } }
  • E.g. Linq from s in studentList group s by s.StandardID into sg select new { sg.Key, sg };
  • Collection Pipeline - "organize some computation as a sequence of operations which compose by taking a collection as output of one operation and feeding it into the next."
    • e.g. method chaining: students.filter(...).map(...).distinct(...).count()
    • e.g. nested function: count(distinct( map (... filter (...))))
  • Collection pipeline:
    • Data Source --> Intermediate Operation* --> Terminal Operation
    • May have many intermediate operations.
    • After the terminal operation we cannot chain any intermediate operation.
  • Query Example: From a students listing select the first surname starting with letter A of a student with number greater than 47000.
  • Divide and reorder requirements:
    • From a students listing... - data source
    • ... of a student with number greater than 47000 - - intermediate operation
    • ... surname ... - intermediate operation
    • ... starting with letter A... - - intermediate operation
    • ... select the first... - terminal operation
  • Kotlin 2 distinct APIs for collections pipeline: Iterable versus Sequences
  • Collections are eager with horizontal processing
  • Sequences are lazy with vertical processing and can be infinite.
  • Marble diagrams.
  • Implementing convert and where equivalent to the eager version of map and filter.

Lesson 27 - yield

  • Implementing convert and where equivalent to the lazy version of map and filter.
  • Implementing lazy iterators SequenceConvert and SequenceFilter
  • sequence - Builds a Sequence lazily yielding values one by one.
  • suspend fun yield(value: T) - Yields a value to the Iterator being built and suspends until the next value is requested.

Lesson 28 - Closeable

  • The try-with-resources Statement
  • A resource is an object that must be closed after use.
  • Kotlin use{} extension
  • Cleaner - manage cleaning actions.
    • public Cleanable register(Object obj, Runnable action) - Registers an object and a cleaning action, i.e. Runnable
    • Cleanable has a single method clean() that runs the cleaning action and unregisters the cleanable.
  • Explicitly invoke the clean() method when the object is closed or no longer needed.
  • If the close() method is not called, the cleaning action is called by the Cleaner.

Lesson 29 - Exercises with lazy sequences

  1. fun <T> Sequence<T>.concat(other: Sequence<T>) = Sequence<T>
  2. fun <T : Any?> Sequence<T>.collapse() = Sequence<T>
  3. fun <T> Sequence<T>.window(size: Int) = Sequence<Sequence<T>>

Lesson 30 - Generics and Type Erasure

  • Type Erasure:
    • Replace all type parameters in generic types with their bounds or Object
    • Insert type casts (i.e. bytecode checkcast) if necessary to preserve type safety.
  • Type Erasure and the need of parameter destination: KClass<T>
  • Inline functions and Reified type parameters to allow the use of T::class
Clone this wiki locally