Weekly Report

Week of

Goals for the week

  • Figure out how to do fault injection with JCM

Daily Progress

  • Monday: Tried to read EMR but was unable to, in hindsight it only report bits that effects the design so faults might have been injected at the wrong places for it to work.
  • Tuesday: Worked on connecting the dev board to the sniffer. Originally had problems with the software sometime not reconizing the JTAG chain. After working at the problem, the usb blaster has two ground pins and both of them need to be connected to the dev board for the software to reliability reconize the JTAG chain. This allowed the system console to work. Was unable to find the fault information from EMR read by searching directly for address. Did some bit shifting and found a location where it might be.
  • Wednesday: Injuected faults in different locations to see how it affected the possible location. It was the right location and was able to determine which bits goes to which field. There are others bits that are set but it curretly unknown what they do. Applying what I learned, was able to find where the address are located when injecting faults.
  • Thursday: Tried to find more patterns in the sniffs. In the large dr scan sections where the fpga in in BYPASS, eventually the TDO become the the TDI but left shifted by one.
  • Friday: Found what looks like what needs to be shifted in during 0x201 instruction to perform a fault injection or read the EMR. For the most part injection faults or reading the EMR is the same, but there are differences and the reasons for the differences are unknown.

Week of Auguest 9, 2021

Goals for the week

  • Document findings so far
  • Start sniffing communication with the SDM

Daily Progress

  • Monday: Updated documentation to include findings on how to program the radiation board. Started looking into how to do fault injection using the JCM
  • Tuesday: Sniffed fault injections centered around a bit and then changed it by one on both ways for bit and frame (the sector can also be slected but was not used at the time since it was not known). Due to the command line to take a while to start up, it took multiple attempts to collect this data. Also did not reconfigured the board after each fault injected so there might be variation between switching the bit from 1 to 0 or vice versa.
  • Wednesday: Changed the sniffer software to stop collecting when recieved a SIGINT signal instead of timing out. This also seems to make it more reliable with preventing overflow but not have been testing so cannot be sure oneway or another. When a lot of data being sent, 129 bits are shifting into the IR register (the stratix 10 IR register is only 10 bits). There is no indication that anything modifies the JTAG chain after configuration. In stratix 10 project, it produces different bitstreams depending if the enable SEU/fault injection was selected.
  • Thursday: There seem to be some device(s) in the JTAG that needs to have it(s) IR register set by shifting in 0x0FECC0FF. In previous version of Quartus, an fault injection IP needed to be included in the design to be able to use the fault injection debugger, but the IP is not listed in available IPs that we have. Intel does require a license to use the fault injection debugger so it is possible that jtag stream is encrypted. Fault injected the same bit multiple times to see if there is any varibility between fault injections.
  • Friday: Did not come in.

Week of August 2, 2021

Goals for the week

  • Get JCM to be able to program Statix X

Daily Progress

  • Monday: Got a new carrier card since the previous one had a bad electrical connection for the TDO pin (always read high).
  • Tuesday: Worked on reading the JAM file
  • Wednesday: Worked on reading the JAM file. Couple of things of note
    • The first two sets of 0x201 and 0x202 is to synchronize the FPGA and the programmer
    • The second set of 0x201 and 0x202 is to request configuration of the FPGA
    • The first 0x208 is to get ready to send data
    • The proceeding set of 0x002 and 0x208 is use to send data and to check if it was all received correctly
    • If during the check performed during 0x208 return a value where the top five bits and the bottom bit are non-zero it will not attempt to fixed but keeps checking until something causes it to time out.
  • Thursday: Tried programming using the JCM running at different clock rates. They always took around the same of time for the program to terminate. Looking at the parsed output, they always fail around the same location. When also considering that the SDM was not present on the Stratix 5 where the JCM previecly worked, it lead to the hypothesis that the confiuration might be timing out. Worked on changing the code for the jam player so it could send up to 32 bits at a time.
  • Friday Worked out the bugs that was causing the updated version from getting to the data section of the JAM file. Once able to get to the data section, there was about a 14x speed up when sending configuration data. This was enough to allow the JCM to be able to program the Stratix X.

Week of July 26, 2021

Goals for the week

  • Get JCM to be able to program Statix X

Daily Progress

  • Monday: All that Intel sequences were different and got shifted in instructions that seems to be errors and I thought it might be because of the logical level differences. Was able to get the level shiffter to work (don’t know how) and collected four more programming sequences using the level shiffter. Unfortunately the level shifter did not remove the problems. Started to compare the 8 files and started compiling what seems like what the the JAM file wants if there are no problems.
  • Tuesday: Finished compareing the differences between the sniffed sequences. Changed the sniffer to sample TCLK instead of being directly triggered by it.
  • Wednesday: Adjusted testbench as it was not changing the JTAG signals when TCLK went low. Changed some of the tests to account that there is a minimum and maximum duty cycle for TCLK to make sure the transition is latached in the flip-flops. Added a less verbose output for the sniffer parser and made it the default output.
  • Thursday: Did not come in
  • Friday: Collected some data using the new sniffer and it seems like the sniffer is giving more consentant output. It might be that the sniffer is still not perfect or that there is some slight variation between runs (for example verifying that the last section of configuratoin data was recived correctly). The application code for the sniffer frequently lets the FIFOs overflow even when the intel part programs at 500KHz and FCLK2 is set at 50MHz, and this seems to be corrolated with a DMA read error.

Week of July 19, 2021

Goals for the week

  • Talk with Dr. Wirthlin about next steps
  • My second goal
  • My third goal

Daily Progress

  • Monday: Started to look at the difference between Intel and JCM sequences. They are largely the same, except there seem to be a good chunk missing from the JCM sequence.
  • Tuesday: Worked on moving all relative files to the sniffer git repo and made sure all the build scripts worked correctly
  • Wednesday: Fixed a makefile that was preventing compiling the sniffer application code. Collected data on the JCM JTAG sequence.
  • Thursday: Sniffed Intel JTAG sequence four times. Parsed all generated data and started to compare the Intel sequences.
  • Friday: Pioneer Day

Week of July 12, 2021

Goals for the week

  • Update sniffer parser to use three seperate files instead of one

Daily Progress

  • Monday: Adjusted parser to use three input files instead of one. Use able to get the sniffer to not overflow when Intel usb blaster was running at 24 MHz, however it does not seem to be able to provide good data as the parser did not reconize instruction codes that was expected (for example it did not reconize the ID code instruction). It is possible that it is becuase the sniffer is not getting a code square wave for TCLK. Slowing the usb blaster does seem to provide good data.
  • Tuesday: Used the JAM player on the JCM to collect some of data of it trying to program the Sandia board using the debug option. Created a python script (do we want to save it somewhere on github?) to convert the text file from the JCM to binary though it requires some human edits on the text file. The easist and quickist way is to use the find and replace feature in a text editer. Using a difference checker, compared the two parsed outputs and could not find anything differenes that would obviously cause any problems.
  • Wednesday: Decided to let the JCM run while reading some Intel documentation. VCCE kept on tripping over-voltage protection. From the documentation, Stratix 10 has a max JTAG frequency of about 33MHz (30ns). Worked on trying to understand the jam file that is being used to try to program the board.
  • Thursday: Did not come in
  • Friday: Did not come in

Week of July 5, 2021

Goals for the week

  • Get sniffer to correctly record JCM’s JTAG module transmissions
  • Test Sniffer on Intel USB JTAG programmer

Daily Progress

  • Monday: None, 4th of July observed
  • Tuesday: FCLK2 is not enabled by default, need to enable it through the command line. Its a similar process to setting is frequency, but set enble to a none-zero value. By sending 32 bits and trying multiple values for TDI, it seems like TDI is being used as the clock signal instead of TCLK.
  • Wednesday: The voltage translator seems to not work well with the JCM, sometimes it prevents it from being able to read the ID Code from the Intel FPGA. After trail and error, was able to repoduce the problem that TDI was bing used as the clock signal. Switching the two signals, the sniffer was able to correctly count the number plus one since JCM rising TCLK high before starting any transactions.
  • Thursday: Turns out that the version of the JCM board that is being used, the TCLK and TDI signal was switched. By switching the wires, the signals were able to be recorded expect for TDO. It always read as a logical ‘1’ even when connected to ground. It seems like it is a bad connection so without getting a new JCM board. Since previous attempts at using Intel’s USB Blaster 2 caused the FIFOs to over, started testing with JCM’s burst transfer to minimize the amount of cpu usage needed to send large amount through JTAG. Since this also caused overflow, changes were made to the driver to remedy it. Switched back to Intel hardware and it still instantly overflow the FIFOs.
  • Friday: Removed most of the debug print statements, both in user and kernal space. This allowed sniffing of intel signal up to 6MHz. The FIFOs are able to mostly keep up at higher TCLK rates, but at some point there is a longer delay between when the FIFO indicates that it is above its interrupt value and when the DMA starts reading data out of the FIFO causing it to overflow. Some possible reasons: the kernal is busy handling higher priotiy interrupts or the userspace code that starts the DMA read is momentarily not the process being run.

Week of June 28, 2021

Goals for the week

  • Get TB working for multiple clock rates and duty cycles of TCLK
  • Verify the sniffer is working in hardware on a signle board
  • Test sniffer with JTAG signals coming from seperate board

Daily Progress

  • Monday: Finished writting testbench and all test are passing. Added read file operation to sniffer driver to block until interrupt.
  • Tuesday: Verified the ability to write correct values to files. Started to refactor test code into usable application code.
  • Wednesday: Finished writing code for application code. Modify block diagram so the sniffer use external JTAG snigals. Soldered headers onto the breakout board and splice in a female connector unto exisiting male-female jumper wire.
  • Thursday: Wire everything togather in order to start sniffing external JTAG signals. Something seems work with TCLK since it is only registering 4 clock cycles on TCLK. Doubled checked all the connections, but failed to find any problems.
  • Friday: Switched to using the JTAG module on the JCM to try to track down the reason why the sniffer was not working. Added FCLK2 to the JCM so the sniffer would use a seperate JCM clk then the module uses. Still unable to get the sniffer to work so decided to create a counter that used TCLK as the clock enable to make sure the TCLK signal was getting into the FPGA.

Week of June 21, 2021

Goals for the week

  • Finish Code for JTAG sniffing
  • Implement mechanism to get last incomplete word into FIFO
  • Test sniffer with JTAG signals coming from seperate board (need to wait for sparkfun part)

Daily Progress

  • Monday: Seperated test code into different threads for sending JTAG and sniffing JTAG lines. This seems to generally improve expandability of adding more test cases.
  • Tuesday: Worked on verifying being able to read multiple words from the FIFO with the DMA. Need to be carefull is it possible to get the DMA stuck in a bad state that requires the JCM to be rebooted to clear the issue.
  • Wednesday: Worked on adjusting the sniffer code to run on the JCM clock domain instead of the TCLK domain. This will allow most of sniffer registers to be clocked at a reliable rate.
  • Thursday: Started to reworked the testbench to reflact the new timings. SD card boot partition got currupted so need to re-image to fix it.
  • Friday: Still try to get the timing on the testbench correct. Started to change how the timing in the testbench so it did not rely on TCLK.

Week of June 14, 2021

Goals for the week

  • Finish Code for JTAG sniffing
  • Implement mechanism to get last incomplete word into FIFO
  • Test sniffer with JTAG signals coming from seperate board (need to wait for sparkfun part)

Daily Progress

  • Monday: Worked on debugging code. Found bug in driver (also found in JTAG driver that needs to be fixed) where it was coping value to another kernel space variable instead of user space variable.
  • Tuesday: Worked to get SD card boot patition fixed, created an backup image to make fixing in the future quick and painless.
  • Wednesday: Struggle tracking down and fixing bugs, did not make much progress.
  • Thursday: Figured out that the DMA was not accessing the sniffer FIFOs because wrong device ID in sniffer driver.
  • Friday: Got TCLK counter working. Since TCLK is not a free running clock, the counters that run on it needs to have an asynchronous reset.

Week of June 7, 2021

Goals for the week

  • Write driver code for sniffer
  • Test sniffer in hardware

Daily Progress

  • Monday: Studied existing JTAG driver to see what was able to be reused in Sniffer driver.
  • Tuesday: Can’t recall specifics, something to do with writing code for the sniffer
  • Wednesday: Can’t recall specifics, something to do with writing code for the sniffer
  • Thursday: Wrote userspace code that interrects with the driver
  • Friday: Started to write code to implement sniffing. Encountered a bug where depending on where functions that read from the sniffer registers, different values where return when trying to read the same registure.

Week of May 31, 2021

Goals for the week

  • Fishish writing testbench for Sniffer
  • Write driver code for sniffer
  • Test sniffer in hardware?

Daily Progress

  • Monday: None, Memorial Day Holiday
  • Tuesday:
  • Wednesday: Added long real world test to testbench.
  • Thursday: Added final edits to testbench. Integrated in exisitng JTAG design
  • Friday: Got sniffer bitfile to work on JCM. Cleaned up Sniffer file orginization.

Week of May 24, 2021

Goals for the week

  • Finish Sniffer HDL (need to work on DMA connection)
  • Verify that Sniffer HDL is working currectly
  • Write Sniffer driver

Daily Progress

  • Monday: Worked on finishing Sniffer HDL. Accidenlty deleted work when working on cleaning up files
  • Tuesday: Quickly recreated work from previous day. Started to look at how the JTAG module testbench worked.
  • Wednesday: Copyed testbench into sniffer project. Had trouble has it was showing syntax errors in my project where they were not showing in the jtag project. Started to work on work arounds to get around the syntax error. Started searching to find reason for one project having syntax errors while the other did not.
  • Thursday: Found out that the testbench in the jtag project was set to VHDL 2008 while the testbench was set to VHDL in the sniffer project. Started to make modifications to the testbench to match what needs to be tested in the sniffer. Was unable to get the timing right for checking a single word, the testbench always wanted what it would be the following TCK cycle.
  • Friday: Added more test to the testbench while thinking why the off by one cycle problem. Turns out it was not accounting for the signle to be latched so it needed one more clock cycle. Unable to get it working when reading multiple words. For each word, the drift between what was received and what was expected increased by one (first word was correctly time, second word was a cycle off, third was two cycles off, etc.). Out of curiosity, synthesized the sniffer to see if it will impley the existance of a clock buffer for TCK, it does.

Week of May 17, 2021

Goals for the week

  • Understand applicable JCM code
  • Create parser for JTAG Sniffer output

Daily Progress

  • Monday:
    • Created first draft of parser specs
    • Started writing code for parser
    • 20.4 JAM file ~35KB larger then 20.1
      • Unable to check if it is do to quartus.ini as admin privileges are needed to modify the file
      • JCM does not seem to be able to program the board is it does not terminate
  • Tuesday:
    • quartus.ini does change the jam file
    • jam files generated with 20.4 work with quartus_jli 20.1
    • Does not appear there is a way to get JTAG signals from intel programs directly
    • Parser largely functional. Need to add more logging and verbose outputs.
  • Wednesday:
    • Minor refactoring of parser code to make it eaiser to read
  • Thursday: Started to create sniffer vhdl code using existing JTAG IP as a base.
  • Friday: Continue work on sniffer vhdl code

Week of May 10, 2021

Goals for the week
  • Reorginze and improve project wiki
  • Get JCM to work reliably on Dev Board
Daily Progress
  • Monday: Took a closer look at troublesome output from dev board. Seems like JCM is not reciving the signals some time and gets out of sync other times. Started to look at JCM Jam and JTAG code.
  • Tuesday: Work on getting Linux enviroment set up. Looked at JCM code.
  • Wednesday: Try to debugg issue between JCM and dev board. Still unable to determine what is causing the problem where JCM unreliably reads the ID Code.
  • Thursday: Able to get Sandia board programmed using Sandia power settings. Note their work around only works with Quartus Prime Pro 20.4. Also did work on improving the wiki. Since our install is on the Windows partition, seems like I need to do most of my work in Windows instead of Linux.
  • Friday: Since the Sandia board is the focus of the current project, the decision was made to put on hold on getting the JCM to work on it. Continued to look at the JCM code. Worked on improving the project wiki, all the old wiki files are found on the old_wiki branch.

Week of May 3, 2021

  • Monday: Worked on understanding JCM, getting data out of design with JTAG
  • Tuesday: ???
  • Wednesday: Tried to get design to fail with fault injection. Attempted to program dev board with JCM. Was unable to get past IDCODE check.
  • Thursday: Able to get IDCODE from radiation board using the JCM. Able to use quartus JAM player to program dev board. Unable to get JCM to work with dev board. Not sure if connecting to it correctly.
  • Friday: Continued to try to get JCM to work on Dev board. Worked on improving project documentation/wiki
Report template

Week of

Goals for the week

  • My first goal
  • My second goal
  • My third goal

Daily Progress

  • Monday:
  • Tuesday:
  • Wednesday:
  • Thursday:
  • Friday: