Joint PSP/Taskforce minutes for 10 Sep 2020
============================

(a) Kiyomi will give a talk on DT replacement:
  (1) Many people worked on this project.
  (2) Vacuum leak on DT #2 in Tank 2. Downstream of a large manifold. It is a very crowded area.
  (3) History of DT replacement. 11 DT since 1985. 3 DT in last years.
  (4) After DT26 in Tank3 was replaced in 2018 losses was significantly reduced.
  (5) How we rebuild DT. DT body, endcaps and bore tubes.
  (6) We originally had 4 spares, and not all usable. They do not match the
  required replacements. We had to use the exhibit DT for Tank2 DT2
  replacement
  (7) Process of removing quad required cutting and cleaning sticky
  compound inside that holds quad.
  (8) 15 Hz pulse test to check heat transfer. Pulsed magnet at 250 A
  with different water pressure and temperature.
  (9) Attached endcap using ebeam welding.
  (10) Final length +/- 2 mils.
  (11) Leak checked and 15 Hz pulse test for 2 weeks.
  (12) How to install
    (i) First to remove tank 2 end cap.
    (ii) Rad survey for each DT. Alignment precision +/- 5 mils. Measured
    upstream and downstream position.
    (iii) Removal DT2 is very crowded in the area + post couplers . Used
    2D and 3D model to make sure that the DT can be removed.
  (13) Position and radiation survey as found. Horizontally looks
  aligned. Vertically there is a sag. Rad survey is high at the
  end. DT60 is tilted which is causing this. Realigned DT60 and DT11.
  (14) Installation and alignments took 6 days.
  (15) We now have alignment data for Tank 2, 3 and 4. Tank 1 has slope
vertically.
  (i) We plan to move the PreAcc vertically to so that beam goes
  straight into angled Tank 1.
  (16) We have to build from spares from scratch. 

(b) Brian talked about the status of the GMPS LDRD
  (1) GMPS LDRD is a pilot project for implementing ML in the
  accelerator. Other reasons to do this:
    (i) Hard to get spare parts for present system.
    (ii) GMPS good fit for ML because there are many errors from the
    envornment, e.g. HLRF power lines and mode of operations.
    (iii) Tighter regulations required in PIPII era.
  (2) Regulation of IMIN requires anticipation of errors, account for
  disturbances  minimize steady state IMIN.
  (3) Sources of IMIN errors comes from
    (i) variations in line freq,
    (ii) sag from ramping MI,
    (iii) gallery temp
  (4) The project starts with
    (i) Characterization of the current VXI system.
    (ii) Setup 15 Hz dataloggers.
    (iii) Grab a state of the system so that we can work from this
  data.
  (5) Challenges are that we don't really want to change the
  operational system.
  (6) Timing issues with dataloggers.
  (7) From the analysis, line frequency 60 Hz definitely plays a role
  in the error.
  (8) Using data to train a ML reinforcement model.
  (9) Studies during a PS study day.
     (i) Ramped down the proportional and integral gains to zero.
     (ii) Lower gain, error is larger but did not see any improvement
     by increasng gain by 20%.
     (iii)  See plots
     (iv) RF loads the line, bias supply draws a lot of difference. The
     line frequency drifts. As the MI ramps, we see an effect. Nice to be
     able to remove.
  (10) WE have a FPGA model calculates the B-field needs to be and
  sent to PS.
  (11) ARRIA 10 SoC will be used in this project.
  (12) HW was a repurposed a board that Brian C. and Philip V. designed.
    (i) Presnetly, the ADC input range is too small: 1-2 Vpp. Expected
    transductor voltage is 0 to 10V. 12V.
    (ii) Brian has the board and is being checked out. Peter has
    powered it up and found one issue.
  (13) Eventual upgrade is to remove all the NIM modules except
  the. zero crossing detector.
  (14) How to integrate ML into legacy system will require rigid
  safety requirements.
     (i) Need to think about putting in constraints. Integrate ML to
     traditional PI control. No diode clamps. Will use legacy system
     to check the limits.
  (15) Reinforcement learning explained.
    (i) Model will figure out the functions in high dimensional
    systems.
    (ii) With a reasonable model, we can reward actions that minimize
    steady state  errors.
    (iii) Example inputs:  Bfield measurements with transductor
  (perhaps voltage divider), temperature  etc. maybe losses in the
  future.
  (16) Implementation takes big pile of data transformed into an ML
  model. Then take the ML model use into HLS4ml framework which spits
  out C++ code. Stick this into HLS compiler that transforms C++ code
  to Verilog. However, using HLS has challenges.
  (17) Rack space reqruiements is 2U and PS. This will include a local
  and training server.
  (18) FE software: MOOC is dead. Will need an ACNET shim layer or
  EPICS FE.  One thing nice thing about using Linux is ability to use
  standard programmes to make APIs.
  (19) Garbage data can be problem. The problem is that bad data could
     have been used to train the model.
     (1) Example: MADC readback IMIN error had a cable issue.
  (20) Extensions to include beam loss as an input. difficult because
  beam loss may not be related to GMPS..
    (i) Dynamic online training of parallel RL agent. How to back out
  if last few corrections are not good.
    (ii) Can GMPS communicate with cogging or LLRF in a meaningful
  way?
  (21) Timelines:
    (i) Board is on hand.
    (ii) Firmware being developed.
    (iii) Train model to use historical data.
    (iv) Commissioning: digitize signals and comparing model output to
    operational program. 

What signals do you want? Start preparing for you. Touch base. Presently we already have outputs in parallel already. Doris fixed up a shelf ready for a new FE. Some panels for signals