TOPS Lab Logo

Wisconsin Traffic Operations and Safety Laboratory

The WisTransPortal System

The WisTransPortal system serves the computing and data management needs of the Wisconsin Traffic Operations and Safety (TOPS) Laboratory. The project scope includes support for ITS data archiving, real-time traffic information services, transportation operations applications, and transportation research. Learn more.

Outages and Scheduled Downtimes
Return to Main Status Page
Complete Status History (Last Modified Sun, 03 Oct 2021, 03:33:27 PM):

2 0 2 1

  • On October 03, WisTransPortal server maintenance window is scheduled from 6am to 3pm. Maintenance includes Windows Updates and Patches, Qualys recommended security fixes, VMware Patches, and Firmware updates. Server reboots will occur throughout the maintenance window. ~ COMPLETED
  • On September 12, there will be a brief maintenance window 9am to 9:30am. Maintenance includes a brief outage for the WisTransPortal Database. Outage is expected to last no longer than 10 minutes. Applications affected include Crash Data and Reports, TIA, LCS, and InterCAD. ~ COMPLETED
  • On August 29, WisTransPortal server maintenance window is scheduled from 6am to 3pm. Maintenance includes Windows Updates and Patches, Qualys recommended security fixes, VMware Patches, and Firmware updates. Server reboots will occur throughout the maintenance window. ~ COMPLETED
  • On August 2, the following video tours are temporarily unavailable due to system maintenance.
    • SE Media
    • NW Eau Claire
    • SE Project
    • SE Waukesha Co
    • NW Region
    • SW Dane Co
    • SW Project
    • SE Washington Co
    ~ COMPLETED
  • On July 25, WisTransPortal server maintenance window is scheduled from 6am to 3pm. Maintenance includes Windows Updates and Patches, Qualys recommended security fixes, VMware Patches, and Firmware updates. Server reboots will occur throughout the maintenance window. ~ COMPLETED
  • On July 30, The TOPS Lab has a low-risk maintenance window on Wednesday, June 30, from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement Resolve System enhancements. This maintenance will also address an issue that was discovered after last week’s June 23 maintenance update. The maintenance will be limited to the Crash Resolve System. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED
  • On June 27, The TOPS Lab has a maintenance window from 6am to 9am, to perform an update to the VMware infrastructure supporting the Crash Oracle database. This maintenance will require shutting down the Oracle Database during the process. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will be impacted. ~ COMPLETED
  • On June 23, the TOPS Lab has a low-risk maintenance window from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement Resolve System enhancements for two JIRA cases pending acceptance by the CRU in the TOPS Lab UAT environment. The maintenance will be limited to the Crash Resolve System. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED
  • On June 20, WisTransPortal server maintenance window is scheduled from 6am to 3pm. Maintenance includes Windows Updates and Patches, and VMware Patches. Server reboots will occur throughout the maintenance window. ~ COMPLETED
  • On June 8, one of the 8 port LINK decoders (receiver station) is experiencing problems. A trouble ticket has been opened with Teleste to address but ETA to repair is unknown at this time. ~ Closed on 6/11 9:00 AM
  • On May 25, one of the 8 port LINK decoders (receiver station) is experiencing problems. A trouble ticket has been opened with Teleste to address but ETA to repair is unknown at this time. The following video is affected:
    • LINK SE Media
    • LINK NW Eau Claire
    • LINK NW Region
    • LINK SE Project
    • LINK SE Washington Co
    • LINK SE Waukesha Co
    • LINK SW Dane Co
    • LINK SW Project
    ~ Closed on 6/1 6:30 AM
  • On May 23, WisTransPortal server maintenance window is scheduled from 6am to 3pm. Maintenance includes Windows Updates and Patches, Qualys recommended security fixes, Tomcat updates, Web server network fixes, PDC fixes, Firmware updates, and Java updates. Server reboots will occur throughout the maintenance window. ~ COMPLETED
  • On May 9, there will be a Campus-Wide planned outage that will occur as a result of major network maintenance. The window is 12am until approximately 2am (around 2 hours). ~ COMPLETED
  • On My 15, there will be a Campus-Wide planned outage that will occur as a result of major network maintenance. The window is 11:59pm until approximately 3am on May 16 (around 3 hours). ~ COMPLETED
  • On May 5, there will be a low-risk maintenance window from 9:00 PM - midnight. This is part of the TOPS Lab purge requirement on all crash report records from the Crash Database and Resolve System. This maintenance will not result in any outages to the Crash Database and Resolve System. ~ COMPLETED
  • On April 25, WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ COMPLETED
  • April 22 Due to an issue with some video hardware, the following traffic video feeds will be down:
    • LINK SE Racine\Kenosha Co
    • LINK SW Rock Co
    • LINK SW Tomah\LAX
    • LINK NE Green Bay
    • LINK NE Oshkosh\Fond du Lac
    • LINK NC Wausau
    • LINK NE Appleton\Menasha
    • LINK SE Milwaukee Co
    • ~ Closed on 4/25 6:30 AM
  • On April 21 (moved from 4/14), there will be WisDOT network maintenance on the Core Network between 10 AM and 3 PM. The work is a continuation maintenance to implement AutoRP in the core network from last week. This work is weather dependent. There is a potential for brief outage for Link Video ~ CANCELED
  • On April 7, there will be WisDOT network maintenance on the Core Network between 10 AM and 3 PM. There is a potential for brief outage for Link Video ~ COMPLETED
  • On March 31, there will be WisDOT network maintenance on the Core Network. There is a potential for brief outage for Link Video ~ COMPLETED
  • On March 26, there will be a VSPOC update from 5-5:15 PM. VSPOC will not be available during the duration of the work. ~ COMPLETED
  • On March 21, WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ COMPLETED
  • On March 18, the WisDOT TMC is doing continued network maintenance - BGP Primary Routing Change - from 9AM and ending at 3PM. There will be multiple brief communication outage (up to 15-min) of the statewide ITSNet network (all video traffic, IP devices connected to ITSNet, servers and applications will be unavailable) as the changes are implemented. ~ COMPLETED
  • On March 15,the main Crash Database Oracle server (dot-cr-ora) stopped accepting new transactions due to the recovery area filling up and not allowing logging to continue. It was determined that the old transaction logs are not being cleared off properly, causing the recovery area to fill up. The issue started around 11PM on 3/16 and was resolved on 3/17 @ 7:25AM.
  • On March 16, the WisDOT TMC is doing network maintenance - BGP Primary Routing Change - from 9AM and ending at 3PM. There will be multiple brief communication outage (up to 15-min) of the statewide ITSNet network (all video traffic, IP devices connected to ITSNet, servers and applications will be unavailable) as the changes are implemented. ~ COMPLETED
  • On March 10, the TOPS Lab has a low-risk maintenance window on Wednesday from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement JIRA enhancements 863 (FARS Case Number) and 871 (Resolve Attachments) in the TOPS Lab UAT environment. The maintenance will be limited to the Crash Resolve System. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED
  • On March 3, the TOPS Lab has a low-risk maintenance window on Wednesday, from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement JIRA enhancements 863 (FARS Case Number) and 871 (Resolve Attachments) pending acceptance by the CRU in the TOPS Lab UAT environment. The maintenance will be limited to the Crash Resolve System. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED
  • On February 28, the TOPS Lab has an emergency maintenance window from 6 AM - 12 PM to perform maintenance on the VMware virtual environment software. VMware has released three patches to fix vulnerabilities found in its virtual-machine infrastructure for data centers. Among them is one with a severity score of 9.8 out of 10, that should be patched immediately. There is no planned outage for any application due to the redundancy built into the environment but there is always a possibility with any maintenance. TOPS will monitor but will NOT perform its regular monthly maintenance testing at the end of the maintenance window. ~ COMPLETED
  • February 21, WisTransPortal server maintenance window scheduled from 6am to 3pm. ~ COMPLETED
  • January 27, A low-risk maintenance window on from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement several updates to the Resolve System “Merge Pending Amendments” feature pending acceptance by the CRU in the TOPS Lab UAT environment. This maintenance will be limited to the Crash Resolve System, TraCS Import and DROC services. External web services including IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED
  • January 24, WisTransPortal server maintenance window is scheduled from 6am to 3pm. This should be short since the maintenance will include only Windows updates/patches and server reboots. Packer game @ 2pm! ~ COMPLETED
  • January 20, A low-risk maintenance window from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This maintenance will implement several updates to the Resolve System “Manage Contacts” feature pending acceptance by the CRU in the TOPS Lab UAT environment. This maintenance will be limited to the Crash Resolve System. External web services, including TraCS / DROC / IROC import services, DMV web services and customer purchases of crash reports via the NIC website will *not* be impacted. ~ COMPLETED

2 0 2 0

  • December 20, WisTransPortal server maintenance window is scheduled from 9am to 3pm. ~ CANCELED
  • December 11, a low-risk maintenance window from 7AM - 9AM, to perform a maintenance update to the Crash Database and Resolve System FARS EDT process. This update will implement several new database elements and a bug fix in the FARS EDT web service transmission. ~ COMPLETED
  • December 9, a low-risk maintenance windows from 8PM - 11PM, to perform an update to the Crash Resolve System. This update will implement an enhancement ot the Resolve System "Manage Contacts" feature along with a minor update to the Resolve integration driver restrictions translation table. ~ COMPLETED
  • December 4, V-SPOC will be updated to a new version starting at 5PM. The maintenance window will go till 5:15PM. This version includes the newly developed data quality test results. ~ COMPLETED
  • November 22 WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ COMPLETED
  • November 11 there will be a low-risk maintenance window from 8 pm - 11 pm, to perform an update to the Crash Resolve System. This update will implement several JIRA fixes pending acceptance by the CRU in the TOPS Lab UAT environment. This maintenance will be primarily limited to the Crash Resolve System. It will also include a brief (less than 2 minutes) restart to the import services (TraCS / DROC / IROC). DMV web services and customer purchases of crash reports via the NIC website will not be impacted. ~ COMPLETED
  • November 9 Link Travel times hasn't updated since 7:59 AM. We are working on the issue.

    Link Travel times was back up at 3:02 PM.

  • October 25 WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ CANCELED due to election time change freeze.
  • October 11 WisTransPortal server maintenance window is scheduled from 6am to 9am. New Domain Controller promotion and new certificate install on Crash server. No outage expected during this maintenance. ~ COMPLETED
  • October 5 WisTransPortal server maintenance window is scheduled from 7am to 8am. New Certificate install so a short outage for the web service restart ~ CANCELED
  • October 4 TOPS Lab is having a maintenance window from 6 AM – 9 AM to perform server maintenance to the Crash Database and Resolve System. Maintenance includes upgrading the production web server to the latest version of Apache and creating a request for new certificates. This update will impact availability to external systems including TraCS, Driver Reports (DROC), and WIN/NIC Wisconsin crash reports. The main WisTransPortal website and other hosted applications will not be affected. TOPS will perform its regular testing at the end of the maintenance window. ~ ONLY Certificate request created so no outage
  • September 23 The TOPS Lab has a maintenance window from 8:00 PM – 11:00 PM, to perform an update to the Crash Resolve System. This update will install several modifications to the Resolve system components and database to support the upcoming Badger TraCS 2020 Fall Pack changes. This maintenance will result in brief outages (less than 5 minutes) to the Crash Resolve System and Crash Import web services (TraCS / DROC / IROC). DMV web services and customer purchases of crash reports via the NIC Wisconsin website will not be impacted. ~ COMPLETED
  • September 30 WisTransPortal server maintenance window scheduled from 6am to 3pm. ~ COMPLETED
  • September 13 The TOPS Lab has a maintenance window from 6 AM – 9 AM, to perform an update to the Crash Resolve System in the TOPS Lab UAT environment. This update will install several modifications to the Resolve system components and database to support the upcoming Badger TraCS 2020 Fall Pack changes. This maintenance will results in brief outages (less than 5 minutes) to the Crash Resolve System and Crash Import web services (TraCS /DROC / IROC). DMV web services and customer purchases of crash reports via the NIC Wisconsin website will not be impacted. ~ COMPLETED
  • August 23 WisTransPortal server maintenance window scheduled from 6am to 3pm. ~ COMPLETED
  • August 5 There will be an emergency maintenance today from 10pm to 10:15pm. Some older servers are not completing their backups due to some unstable connections and reboots are necessary to resolve the issues. ~ COMPLETED
  • July 26 DoIT Network Services is planning to perform maintenance on our campus network affecting 432NM, ANIMAL, Core/Cloud, and RBN (Chamberlin Hall and 222 West Washington). This work will take place from 12am-4am. TOPS doesn't anticipate this maintenance to have any affect on the WisTransPortal. ~ COMPLETED
  • July 19 WisTransPortal server maintenance window scheduled from 6am to 3pm. ~ COMPLETED
  • July 19 DoIT Network Services is planning to perform maintenance on our campus network affecting Prod Load Balancers and CSSC Area Upgrades. This work will take place from 12am-4am. This maintenance has the potential to affect networking for the CSSC (central) area of campus, as well as load balanced traffic in the data centers. Impact from this work should be minimal, with anticipated disruptions of no more than 30 seconds. ~ COMPLETED
  • June 30 Link Video pages will be updated with a new player between 11AM and 1PM. Users who run the video will need to refresh thier web pages to get the new player. The current player, Bitmovin, will no longer work starting 7/1/2020. ~ COMPLETED
  • June 21 WisTransPortal server maintenance window scheduled from 6am to 3pm. ~ COMPLETED
  • June 21 DoIT Network Services is planning to perform maintenance on many of the Cisco Nexus switches from 12am-5am. It is expected that each switch will experience three ~ 10 minute periods of disruption in connectivity during this window. Network Services will stagger the updates between HA pairs, to reduce the impact as much as possible. See the full list of affected devices. TOPS Lab does not anticipate any WisTransPortal outage due to this maintenance. ~ COMPLETED
  • June 15 intermittent issues on the crash REST and SOAP Tomcat servers after the planned UW DoIT Sunday network maintenance. The work broke some connection pools in the web servers so failures were occuring until the pools were reset at 3:00PM on 6/15.
  • June 14 DoIT Network Services is planning to perform maintenance on many of the Cisco Nexus switches from 12am-8am. It is expected that each switch will experience three ~ 10 minute periods of disruption in connectivity during this window. Network Services will stagger the updates between HA pairs, to reduce the impact as much as possible. See the full list of affected devices. TOPS Lab does not anticipate any WisTransPortal outage due to this maintenance. ~ COMPLETED
  • June 10 The TOPS Lab has a low-risk maintenance window from 9pm - midnight, to perform an update to the Crash Resolve System. ~ COMPLETED
  • May 31 WisTransPortal server maintenance window scheduled from 6am to 3pm ~ COMPLETED
  • May 20, there will be a low-risk maintenance window from 9pm-midnight, to perform an update to the Crash Resolve System. ~ COMPLETED
  • May 20, there will be a low-risk maintenance window from 7-7:30pm, to perform an update to the TMP System. ~ COMPLETED
  • May 13, there will be a low-risk maintenance window from 9pm-midnight, to perform an update to the Crash Resolve System. ~ COMPLETED
  • May 10th UW Network Services is planning to perform maintenance on the data center hardware firewall from 5am-6:30am on Thursday. There is no anticipated disruption to the network during this maintenance.

    See Outage #10590 for a complete list of affected services

    This work will introduce a patch to fix an intermittent bug that has been causing disruptions on UW firewalls recently. ~ COMPLETED

  • April 29 There will be a TIA maintenance on Wednesday, between 8:00 PM and 11:00 PM. TIA will not be available for <15 min during the maintenance window. ~ COMPLETED
  • April 26 WisTransPortal server maintenance window scheduled from 6am to 3pm ~ COMPLETED
  • March 22 WisTransPortal server maintenance window scheduled from 6am to 10am ~ COMPLETED
  • March 9 TOPS planned an emergency maintenance on the WisTransPortal Web server to fix an network issue. The WisTransPortal was unavailable for 6 minutes total between 11:32pm and 11:56pm.
  • March 4 A low-risk maintenance window, from 9pm-midnight, to perform an update to the Crash Resolve System. This maintenance window will implement several updates and improvements to the Resolve System “notice to report” letters along with several other minor JIRA fixes. ~ COMPLETED
  • March 2 There was a DoIT network issue that took down access to the WisTransportal and all its applications. The outage started at 11:40 am and was resolved at 12:53 pm. Campus Outage Link
  • February 23 WisTransPortal server maintenance window is scheduled from 6am to 3pm. ~ COMPLETED
  • February 5 there will be a Teleste Receiver Station Update beginning at 2am and ending at 6 am. This work is to alleviate an issue with the Receiver Stations keep crashing (issue #22). During the update, one of the LINK server will not be available, which means that the few of the video tours will not be available in LINK for short period of time. ~ COMPLETED
  • February 5 A low-risk maintenance window is scheduled from 9pm-midnight to perform an update to the Crash Resolve System. ~ CANCELED
  • January 26 WisTransPortal server maintenance window scheduled from 6AM to 3PM. ~ COMPLETED
  • January 14 There will be a Teleste Receiver Station Update beginning at 2am and ending at 6am. During the update, the LINK server will not be available, which means that some video tours will not be available in LINK for short period of time. ~ COMPLETED
  • January 12 DoIT Network Services is planning to perform maintenance 12am-8am on the data center hardware firewall that will have a significant impact on a number of DoIT-hosted services. The scope of impact for this work includes many systems, including the WisTransPortal. Some systems may only experience disruption for a few minutes, but others may be impacted for an hour or more while system and application admins bring them back online, if necessary. DoIT list of affected services ~ COMPLETED
2020
Scheduled WisTransPortal Server Maintenance Days Completed Notes
December 20 CanceledPLANNED: Servers patched up to Microsoft December Updates; Skip CRASH Resolve servers
November 22 Completed Servers patched up to Microsoft November Updates;
October 25 CanceledServers patched up to Microsoft October Updates; SolarWinds troubleshooting and upgrade;
September 20 CompletedServers patched up to Microsoft September Updates; LINK Video pages update - new tours;
August 23 CompletedServers patched up to Microsoft August Updates; SolarWinds update
July 19 CompletedServers patched up to Microsoft July Updates; VMware updates;
June 21 CompletedServers patched up to Microsoft June Updates; Server certificates
May 31 CompletedServers patched up to Microsoft May Updates; firmware updates;
April 26 CompletedServers patched up to Microsoft April Updates; VMware updates; SFTP Storage increase; SolarWinds Update
March 22 CompletedServers patched up to Microsoft March Updates;
February 23 CompletedServers patched up to Microsoft February Updates; Wowza Update; VMWare patches and upgrades;
January 26 CompletedServers patched up to Microsoft January Updates; Dell critical firmware updates; Apache updates; VMWare patches and upgrades;
January 12 CompletedDoIT Network Services maintenance on the data center hardware firewall

2 0 1 9

  • December 22 WisTransPortal server maintenance window scheduled from 6AM to 9AM. ~ COMPLETED
  • December 8 WisTransPortal server maintenance window scheduled from 6AM to 9AM. ~ COMPLETED
  • December 4 DANE InterCAD server move had some issues so DANE InterCAD was down from 1:00pm to 1:45pm
  • December 4 DANE InterCAD server move from 12:00pm - 12:15pm ~ COMPLETED
  • November 28: @ 3:03AM the virtual environment storage, 1 volume filled max reserved space and caused some systems to go into standby mode. This affected some automated services to include Crash Resolve and Winter Roads 511. The storage was corrected around 9:40AM. Systems recovered at 11:04am and all is running well at this time.
  • WisTransPortal server maintenance window scheduled for Sunday, November 24 ~ COMPLETED
  • November 6 11:45 AM - UW Network firewall started experiencing issues starting @ 11:18 PM on Nov 5. This event slowed connection to the WisTransPortal and all applications Campus Outage. UW DoIT Firewall failed to secondary and things came back up. This caused some stale sessions and prolonged outages being experienced until about 9am. Systems were good and then the UW DoIT firewall switched to primary and caused WisTransPortal to respond slowly and LINK video to fail starting aroun 9:45am. We are still working to clear LINK video to smooth operation.
    • Nov 7 Update - I continue to have to reset LINK about every 30 minutes. I try to keep up but you may have to wait a bit and then refresh until we get this settled with the DoIT Network team.
    • Nov 8 update - Still dealing with streaming video issues every 30 mintues. No new break throughs.
    • Nov 8 3:30 pm update - actively working on this issue with Palo Alto techs and the UW DoIT Network team.
    • Nov 8 5pm update - Issue resolved.
  • October 29-30 A Teleste update occured from 2am to 6am. ~ COMPLETED
  • WisTransPortal server maintenance window scheduled for Sunday, October 27 ~ COMPLETED
  • WisTransPortal server maintenance window scheduled for Sunday, September 29 ~ COMPLETED
  • WisTransPortal server maintenance window scheduled for Sunday, August 25 ~ COMPLETED
  • 8/22/2019 - The Crash Data Analysis Tools locked up this morning around 9:50am causing the WisTransPortal to be unresponsive. WisTransPortal was back at 9:55am but Crash Data Tools is still unresponsive. It appears to be an issue with the GIS /WISLR map at this time.

    Systems were back at 10:14 AM

  • WisTransPortal server maintenance window scheduled for Sunday, July 21 ~ COMPLETED
  • On July 9, the WisTransPortal became unresponsive between 8:38 AM and 8:55 AM. UW Campus was experiencing a network issue.
  • WisDOT will upgrade the Teleste Video System on the scheduled time below:
    • Early Monday morning, July 8th from 2:00 am to 2:00 pm
    • Early Tuesday morning, July 9th from 2:00 am to 2:00 pm

    During the upgrade, there will be intermittent LINK video outages throughout the process

  • On June 27, the WisTransPortal became unresponsive between 4:08 PM and 4:11 PM. We are looking into the cause.
  • On June 6, 2019, the WisTransPortal became unresponsive starting at 10:04 AM. Services were back by 10:14 AM. We are investigating the issue.
  • On Tuesday, May 28, 5:15 AM, TOPS Lab was notified that half the video was down. One of the servers did stop over the weekend again. This is a re-occuring issue and we are still working on it with Wowza Media support. Video was restored by 7:55 AM.
  • On Wednesday, May 22, emergency maintenance on some WisTransPortal servers. Reboots will have all TMC apps offline briefly. Window is from 9-10PM., ~ COMPLETED
  • Teleste Maintenance work will start late Sunday night, May 5th, 2019 and last until Friday afternoon, May 10th, 2019. There will be intermittent outages with LINK Video throughout the process but they will be limited due to the redundant nature of the system. NVRs will also be upgraded during this process and recordings will be temporarily interrupted for 30-50 cameras at a time as each NVR is upgraded. ~ CANCELED
  • WisTransPortal server maintenance window scheduled for Sunday, April 28, ~ COMPLETED
  • InterCAD maintenance window scheduled for Wednesday, April 24, from 12-12:30 pm. ~ COMPLETED
  • 4/22/2019 6:24 AM - A number of the LINK video tours were reported down. One of the Media servers locked and needed restarting. Tours were restored by 7:32 AM.
  • 4/16/2019 8:55 AM - The WisTransPortal became slow and unresponsive. Web site was back at 9:11 AM. We are looking into the issue.
  • 4/15/2019 12:00 PM - 8 Video streams started failing due to a service error on one of Media servers. There were issues on the backend video systems as well during this time. Service and video was restored by 9:15 PM.
  • 4/13/2019 11:00 PM - Storage maintenance resulted in some stalled services starting at 9:45 AM. We continue to work on the issues. TPIMS, WRS, and LCS updates are some of the updates not going out from TOPS.
    • 4/14/2019 6:32 PM - all systems were back up.
  • 4/13/2019 Storage maintenance from 6AM to 12PM. No systems or applications will be taken down. ~ COMPLETED
  • 4/4/2019 11:14 AM - At 12:04 AM the Dane county InterCAD puller service failed due to DNS issues. The cause is being investigated. Puller was restored at 9:35am
  • VSPOC Calendar update - Sunday March 24th @ 4:00pm. ~ COMPLETED
  • WisTransPortal server maintenance window scheduled for Sunday, March 24, COMPLETED. - NO DOT Crash Resolve (CRU) systems updated or rebooted due to the upcoming election code freeze.
  • TMP 2.0 roll-out on Wednesday, March 20, from 12p-1p. ~ COMPLETED
  • WisTransPortal storage maintenance on March 17. Window is from 10am - 6pm. No production services will be affected. ~ CANCELED
  • CRASH Resolve patch on March 17. Window is from 6am - 9am. Services will be unavailable for short restarts of Tomcat during this time. ~ CANCELED
  • 3/13/09 - At 10:41am, TOPS was notified that V-SPOC was unresponsive. Services had locked up after a search. VSPOC app was restarted and services were back at 10:47am.
  • 3/9/2019 8:20 PM - at 3:15 AM, one of the volumes on the WisTransPortal virtual storage reached capacity and production systems started being unresponsive at times. These servers include the scheduled services, Storm Reports, and Resolve. Issue was resolved at 9:20PM.
  • 3/6/2019 - At 2:54 PM, the WisTransPortal was reported to be responding really slowly by the CRU. This appears to be due to an application they were testing with issues returning information. The application was restarted and systems were back to normal at 3:13pm.
  • Sunday 3/3 at 8am - 9am is emergency maintenance on the main Web server. This will affect all applications when system is rebooting. Maintenance may be canceled due to weather. ~ Completed
  • Friday 3/1 at 11:30am is emergency maintenance for the LINK video system. This is a performance update. ~ Completed
  • on 2/28/2019, the WisTransPortal web server dropped network connectivity between 11:24:24am and 11:35:27am. The issue was caused by system load. Increased resources are being added to the virtual environment to compensate.
  • The 9am - 3pm portion of Sunday, Februrary 24 maintenance, was canceled due to weather conditions.
  • Wednesday 2/20 between 8pm-10pm: LCS update ~ Completed
  • Wednesday 2/20 at noon: LINK update ~ Completed
  • Monday 2/18 at noon: InterCAD update ~ Completed
  • Monday 2/18 at 10am: Recreate RX station for Dane County and SW Tour ~ Completed
  • On February 2nd, some WTP applications became unresponsive around 4:00am. This included TIA and WRS. The issue was with the database being misconfigured after early maintenance on the backups. Sytem was restored at 9:05am on 2/2.
  • Today, January 29, starting at 7:10am, the WisTransPortal became slow and often unresponsive. Things were back at 8:48am. The WTP web server was locked up and needed to restart.
  • WisTransPortal server maintenance window is scheduled for Sunday, January 27 - completed
  • LINK video maintenance on January 3rd, 2019 from 12:30 - 1:30pm ~ COMPLETED
2019
Scheduled WisTransPortal Server Maintenance Days Completed Notes
December 22 CompletedServer reboots - no patching or updates
December 8 CompletedVMWare storage config and VMWare cluster config; New Web server
November 24 Completed Servers patched up to Microsoft November Updates; VMware Updates;
October 27 CompletedServers patched up to Microsoft October Updates; Symantec Updates; Media servers moved to DoIT AD; Storage Firmware Updates;
September 29 CompletedServers patched up to Microsoft September Updates; Symantec update; Firmware updates
August 25 CompletedServers patched up to Microsoft August Updates; New Media server (Media5) to production
July 21 CompletedServers patched up to Microsoft July Updates;
June 23 SkippedPLANNED: Servers patched up to Microsoft June Updates;
June 2 CompletedServers patched up to Microsoft May Updates; vCenter update
April 28 CompletedServers patched up to Microsoft April Updates;
March 24 PartialServers patched up to Microsoft March Updates; SolarWinds update
February 24 PartialDOT CRU Servers patched up to Microsoft February Updates;
January 27 CompletedServers patched up to Microsoft January Updates;

2 0 1 8

  • WisTransPortal server maintenance window scheduled for Sunday, December 16, - completed
  • On December 8th @ 8:45pm, DoIT Network team contacts TOPS about network issues at the University and we are one of the 3 or so other high traffic departments flagged. They want to shut down our traffic for 10-15 minutes to help identify the issue. UW Madison IT Outages
    • Saturday Dec 8 @ 9:30pm(ish) — DoIT team takes WisTransPortal traffic offline for 15 minutes. Steven P and I have communicated with DOT and TMC about the outage and have received the verbal okay.
    • Sunday Dec 9 @ 2:30pm — DoIT network team contacts me to see if we would be willing to move off to a private network. After discussions, DoIT decides this in undesirable and cancels the request. No issues on the WisTransPortal up through this time to my knowledge.
    • Sunday Dec 9 @ 7:15pm — LINK video has stopped. After looking into the issue, it appears that the same issue we had when first migrating to the firewall has returned. Sessions need to be cleared regularly to keep video going.
    • Monday Dec 10 @ 8:06am — The first notice, besides video (6:18am), saying that the WisTransPortal is unresponsive at times – Crash Resolve System (CRU).
    • Monday Dec 11 @ 11:04am — DoIT implements a fix that cuts down the network interface processing from 90%+ down to under 30%.
    • Monday Dec 12 @ 12:30pm — WisTransPortal web site and internal communication seems to have stabilized although some external traffic still has issues.
  • On November 20th @ 12:00 — the DoIT Data Center team worked with TOPS staff and DOT staff in migrating the firewall. This affected all WisTransPortal networks and applications. The maintenance window extended to 5:30 pm with intermittent outages on applications.
    • 11/26 UPDATE. Working through dropped connections through the firewall to include the issues with LINK video, LCX XML, WRS XML, and InterCAD web services.
  • WisTransPortal server maintenance window scheduled for Sunday, October 21, - completed.
  • 10/19/2018 - At 11:03 AM, TOPS was notified that the DMV was having issues importing crash data. After futher investigating, we found that this started at 10:42am and lasted till 12:19. Other IPs were hitting TOPS servers except the mainframe source. It appears that the issue was between the DOT and UW-TOPS.
  • 10/3/2018 - At 9:09 am, initial reports of the WisTransPortal being down were reported on Crash Resolve. The web site was responsive again by 9:20am. Issue is still being investigated.
    Update - 2:25pm / The issue was caused by the main WisTransPortal database being too busy. Project was adjusted so that more resources were left available.
  • WisTransPortal server maintenance window scheduled for Sunday, September 23, - completed.
  • 9/14/2015 - At 12:15 PM, the WisTransPortal Database was restarted to release connections that have been causing issues for a number of weeks. Most applications were unavailable for 8 minutes during the reboot to include TIA, VSPOC, LCS, and TMP.
  • WisTransPortal server maintenance window scheduled for Sunday, August 19, - completed.
    The Wisconsin DOT has core router maintenance was from 2pm - 6pm.
  • On August 15, @ 9:14am, TOPS lab was notified that the WisTransPortal was responding slowly or down. This happened again sometime between 10 and 10:30am. The cause of the slowness has not yet been identified but we continue to investigate.
    Update 8/16: Still looking into the cause of the slowness that was experienced. A ticket has been created with the DoIT Network team. So far, it appears to be a network load issue between clients and WisTransPortal.
  • WisTransPortal server maintenance scheduled for Sunday, July 22, - completed.
  • Please note that WisDOT will perform an emergency network work tonight (Thursday, July 12th) beginning at 7pm and ending at 10pm.
  • Please note that WisDOT will continue to perform network maintenance (fiber splicing) at multiple locations in Madison area. This will possibly affect LINK Video.
    • Monday, June 18, 2018 beginning at 7pm and ending at 5am the following morning
    • Tuesday, June 19, 2018 beginning at 7pm and ending at 5am the following morning
  • WisTransPortal server maintenance scheduled for Sunday, June 17 - Completed.
  • WisDOT will continue to perform network maintenance (fiber splicing) at multiple locations in Madison area. This will possibly affect LINK Video. The work is scheduled during overnight hours on three consecutive nights next week after Memorial Day:
    • Tuesday, May 29th at 7:00 PM to Wednesday, May 30th at 5:00 AM
    • Wednesday, May 30th at 7:00 PM to Thursday, May 31st at 5:00 AM
    • Thursday, May 31st at 7:00 PM to Friday, June 1st at 5:00 AM
  • AT 10:50 PM on 6/5/2018, TOPS was notified that TMC Operators were unable to connect to the WisTransPortal web site. The WisTransPortal virtual environment ran out of space in one of the configured storage pools and many VMs locked up. Some became corrupted so they could not be started again. Web server stopped at 10:22 PM on 6/5/2018. Web server was back online at 4:17 pm on 6/6/2018.
  • WisTransPortal server maintenance scheduled for Sunday, May 20 - completed.
  • 2018-05-09 - At 6:50 AM, TOPS was notified that TMC Operators were unable to connect to TIA. Subsequent reports verified that all WisTransPortal applications were down. Problem was determined to be caused by a UW Firewall acting irregular. Systems were restored by 10:00 AM
  • WisTransPortal server maintenance scheduled for Sunday, April 22, - completed.
  • 2018-04-11 - at 1:00 AM, the DANE interCAD data collector stopped. A required update was pushed out for the VPN client. The update was installed and DANE was working again at 10:00 AM.
  • 2018-04-10 - at 10:30 AM, TOPS staff notice a slowing in performance on WisTransportal and then at 10:48, Web services were down. One of TOPS storage thresholds were maxed out and locked up a number of servers. By 11:08 systems were restored when the storage configuration was adjusted.
  • 2018-04-05 - At 8:35 AM, TOPS determined that the InterCAD Reporting had stopped around 12:40 AM. Services were restarted and system was back up by 8:35 AM. Staff are investigating the issue.
  • 2018-04-02 - At 11:25 AM, TOPS was notified that the WisTransPortal was not responding. It was quickly determined that VSPOC was locked and preventing further communication with the server. After a restart @ 11:35am, applications were available again.
  • WisTransPortal server maintenance scheduled for Sunday, March 25, - completed.
  • 2018-03-15 - At 9:37 AM, Dane County InterCAD XML pull quit working. The VPN client had disconnected again. Connection was restored at 12:00 PM.
  • 2018-03-14 - At around 7:00 AM this morning, Dane County InterCAD XML pull quit working. The VPN client had failed due to a update request. The client failed to update. By 1:30PM, it started working again.
  • 2018-03-12 - At 9:07 AM, TOPS was notified that the LCS map was unavailable. The application serving up ArcGIS for LCS had crashed. At 5:00 PM, it was running again.
  • WisTransPortal server maintenance scheduled for Sunday, February 25 - Completed.
  • 2018-02-19 - AT 4:00 PM it was noted that delayed and dropped connections to data was occurring more and more frequently from WisTransPortal applications. This was affecting most applications including TIA, VSPOC, and InterCAD. A reboot of the database server was done at 1:00 AM on 2018-02-20 to alleviate the issue. A long term solution is under development.
  • WisTransPortal server maintenance scheduled for Sunday, January 21 - Completed.
2018
Scheduled WisTransPortal Server Maintenance Days Completed Notes
December 16 CompletedServers patched up to Microsoft December Updates; Server firmware updates; Oracle hard drives installed; Wowza upgrade
December 6 CompletedLINK Video now displaying all HD video tours starting at 1:00PM
November 20 CompletedWisTransPortal migrated from Cisco to Palo Alto firewall
November 18 SkippedPLANNED: Servers patched up to Microsoft November Updates; ESXi patches; Java Updates
October 21 CompletedServers patched up to Microsoft October Updates; ESXi patches; SSL Certs
September 23 CompletedServers patched up to Microsoft September Updates; Symantec Upgrade; Dell firmware updates;
August 19 CompletedServers patched up to Microsoft August Updates; VMware ESXi Upgrade (1 host);
July 22 CompletedServers patched up to Microsoft July Updates; Oracle Patches;
June 17 CompletedServers patched up to Microsoft June Updates; VMware Fixes from outage; SAN storage fix and updates
May 20 CompletedServers patched up to Microsoft May Updates; Java Updates, vCenter Maintenance; IP updates
April 22 CompletedServers patched up to Microsoft April Updates; Apache 2.4 patches; SAN storage maintenance
March 25 Completed Servers patched up to Microsoft March Updates
February 25 CompletedServers patched up to Microsoft February Updates; VMware ESXi 6 updates; SAN Storage patches
January 21 Completed Servers patched up to Microsoft January Updates; VMware ESXi 5.5 updates;

2 0 1 7

  • Thursday afternoon, October 26th, the WisTransPortal became unresponsive at 1:57PM. the VSPOC application had locked up resources. Services were restarted and the WisTransPortal was back by 2:00 PM
  • WisTransPortal server maintenance window scheduled for Sunday, October 22 - partially completed. SAN virtual volume work did not allow for complete server patching in the alloted maintenance window.
  • Thursday morning, September 14, the WisTransPortal wsa reported to be responding slowly. First report came in around 9:49am on the TMP system. Resolve was reported slow around 10:00am. After combing through the logs, going through all performance history, and testing applications, nothing turned up as cause on any of the applications or equipment that TOPS has control of. We continue to monitor. Everything was responding normally at 10:00am at the University. No other reports outside of the university since 10:21AM.
  • WisTransPortal server maintenance window scheduled for Sunday, September 24 - Completed.
  • Tuesday morning, September 5, the DOT Resolve system was reported to be down starting around 9:06 AM. Further tests showed that all of transportal.cee.wisc.edu was slow and/or unavailable. By 9:15 AM, Resolve was reported running again. TOPS staff are still looking into the cause
  • Tuesday morning, August 29, no new crash data extracts were created. This was due to an expired software license at midnight of August 28. License was updated on production by 9:20 AM. UAT and TEST were done by 11:45 on 8/29.
  • Monday, August 28. At 8:44AM, it was reported that the WisTransPortal became very slow and sometimes unresponsive. At 9:00AM the Apache service was restarted and things were immediately better. We are still looking into the cause.
  • On Tuesday, August 15, the LINK video was scheduled to be down from 8:00 - 8:30PM. Some issues came up during the process and the video was down till 9:30PM - 1hour and 30 min. This was a major security upgrade to the backend media server.
  • On Monday, July 31, many WisTransPortal maps were offline including WisLR Maps, ITS Inventory, and GLRTOC. We are working with some licensing issues and hope to have those back up in the afternoon of Tuesday, August 1.

    UPDATE - still working on the license issue (10:54AM 8/2)

    Maps were back up by 12:14PM on 8/2.

  • On Friday, July 28, around 6:20 am, the WisTransPortal main database had an issue with the log file space. The system was cleared and applications were back up by 7:30am.
  • WisTransPortal server maintenance scheduled for Sunday, July 30 - Completed.
  • The LINK video camera feeds are down due to some persistent network issues. The problem has been on going all week. We expect to have it repaired the late evening of 6/29/2017.
  • The WisTransPortal servers were experiencing delays and slowness around 10:13am this morning (6/21/2017). The delays were related to problems on the MV4000 Crash Data Resources. Systems were restored by 10:45am. Cause is being investigated.
  • 5/1/2017 - > Bulk emails out of WisTransPortal are being delayed in the AM (TIAs). - UW Madison IT Outages
  • WisTransPortal server maintenance scheduled for Sunday, April 30th - Completed.
  • 4/26/2017 - > WisTransPortal applications were experienceing some slowness between 9:50 AM and about 10:30 AM. DOT Crash systems were not affected. Still looking into the cause.
  • WisTransPortal server maintenance scheduled for Sunday, April 23rd - Completed.
  • 4/21/2017 - > WisTransPortal applications were experienceing some slowness between 8:50 AM and about 9:30 AM. We are still uncertain of the cause. VSPOC had locked up some of the WisTransPortal applications. Resolve was still seeing some issues for a little time after VSPOC was cleared. Everything else was fine. An outage is planned for 4/30 to move equipment to new network services for the longterm solution.
  • 4/18/2017 - > WisTransPortal applications are expierencing an outage starting around 7:56 AM this morning. Issue was temporarily resolved around 9:49 AM. It appears to have started up again around 10:45 AM and resolved around 11:18 AM. This ticket/issue closed at 3:30 PM. The problem is the result of one of the 2 primary network switches having extrememly high usage and other issues not yet resolved. This switch has been moved to failover status until new equipment is put in place - planned 4/30/2017.
  • 4/18/2017 - > Bulk emails out of WisTransPortal are being delayed in the AM (TIAs). - UW Madison IT Outages
  • 4/10/2017 - 10:50 AM > Bulk emails out of WisTransPortal are being delayed (TIAs). - UW Madison IT Outages
  • DOT-CRASH web server maintenance scheduled for Sunday April 9 from 6-9 AM. C A N C E L E D - A small application update will occur at this time.
  • WisTransPortal application maintenance scheduled for Wednesday, April 5, from 10 to Midnight, completed.
  • Network issues started again around 10:40 AM on 3/30. Some web applications were unrepsonsive. Network communication restored itself by 5:42 PM on 3/30. Issue was due to heavy traffic on main shared UW Switch.
  • Network issues started around 10:30 AM on 3/29. Web applications were unresponsive. Around 11:10 AM, things appear to be back to normal. Still looking into the issue.
  • WisTransPortal server maintenance completed on Sunday, March 19th.
  • WisTransPortal TEST server maintenance completed on Friday, March 17th.
  • Some services that originate on the DoIT platform have been degraded on March 9, due to an issue with one of the network communications devices. The problem started at 1:40 PM. Resolved at 2:51 PM.
  • The very brief outage today on WisTransportal was noticed around 1:40 PM on Tuesday, February 28. The problem is still being investigated but appears to be due to a UW network issue.
  • VSPOC traffic data was not accessible most of the morning of Friday 2/17. It was back up by 2:20 PM. Problem is being investigated.
  • WisTransPortal server maintenance window scheduled for Sunday, February 19th, was completed.
  • WisTransPortal TEST server maintenance window on February 17th, starting @ 12:00 PM, was completed
  • WisTransPortal server maintenance window was CANCELED for Sunday, December 18th, due to inclement weather. Maintenance scheduled for Sunday, January 22nd, was completed.
  • WisTransPortal TEST server maintenance scheduled for Wednesday, January 18th, was completed. No production or DOT-CRASH servers will be affected
  • VSPOC was down the morning of Jan 9. It may have been down as early as 6:22 PM on Jan 6, 2017. Services were restarted and VSPOC was back online at 9:43 AM on Jan 9.
  • An update to the DOT-CRASH system is scheduled for today, January 6, starting at 8 PM. The window goes till 11 PM. The main WisTransPortal web server will also be restarted during this window.
  • Many WisTransPortal applications went down at 10:03 AM on January 6th. The Oracle Listener service stopped on the main DB server. Server was restarted and services were restored by 10:50 AM.
  • LINK video went out starting on January 2nd at 1:13 AM. The VLC encoders stopped working. Video was back up on January 3rd by 8:15 AM.

2017
Scheduled WisTransPortal Server Maintenance Days Completed Notes
December 10 CompletedServers patched up to Microsoft November Updates; Symantec Update; Wowza Update
November 19 SkippedPLANNED: Servers patched up to Microsoft November Updates; Apache and Apache Tomcat; Java (TEST servers)
October 22 CompletedSome servers patched up to Microsoft October Updates; VMware vSphere configuration changes
September 24 CompletedServers patched up to Microsoft September Updates; VMware ESXi patches;
August 20 CompletedServers patched up to Microsoft August Updates; Oracle patches on test
July 30 CompletedServers patched up to Microsoft July Updates; ESXi patches, VMware tools
June 25 CompletedServers patched up to Microsoft June Updates; Dell Storage firmware and Software updates
May 21 CompletedServers patched up to Microsoft May Updates; Apache and Tomcat patches (OpenSSL) on test, dev and prd; Java Updates on public servers; Security fixes (OCS Scan); New VPC config for Agency Network Connection.
April 30 CompletedWisTransPortal Equipment move in UW DoIT Data center
April 23 CompletedServers patched up to Microsoft April Updates; VMware ESXi patches; Apache and Tomcat patches (OpenSSL) on test and dev; Java Updates; Security fixes (OCS Scan)
March 19 CompletedServers patched up to Microsoft March Updates;Wowza upgrade
February 19 CompletedServer reboots; Agency Network IP changes; New NAT IPs added; Remote Desktop services for servers was fixed.
January 22 CompletedServers patched up to Microsoft January Updates; VMware ESXi patches; Security fixes (OCIS Scan)

2 0 1 6

  • VSPOC was unresponsive on Dec 7, 2016 starting around 10:50 AM. Application was restarted and available at 11:14 AM.
  • WisTransPortal server maintenance window scheduled for Sunday, November 13th, completed.
  • On October 27th around 9:57 AM, the WisTransPortal locked up. After service restarts, the WisTransportal locked up again at 10:05 AM. All applications were unavailable. Services were restarted and things were back up at 10:15 AM. We are not sure what caused the freeze but we are looking into it.
  • The new LINK system upgrade was started on October 24th at 11:30 AM. Due to some Media server problems, it was not published till around 6:00 PM.
  • WisTransPortal server maintenance window scheduled for Sunday, October 23rd, completed.
  • The WisTransPortal was unstable 9/19/2016 starting around 2:25 PM. Applications were offline for up to 2 minutes at a time off and on for 20 minutes or so. This issue has been connected with the WISLR map usage on the MV4000 Crash Data Retrieval Facility. We apologize for the inconvenience. We are working on resolving the issue.
  • WisTransPortal TEST server maintenance window is scheduled for Tuesday, October 18th, was completed on Sunday, October 16th.
  • The Link Beta site was down for about 20 minutes on 9/12/2016 for switch over to private network. It was back up by 2:08 PM.
  • WisTransPortal server maintenance scheduled for Sunday, September 25th, 2016 was completed.
  • WisTransPortal TEST server maintenance scheduled for Tuesday, September 20th, 2016 was completed.
  • The Link Beta site was down for about 20 minutes on 9/12/2016 for switch over to private network. It was back up by 2:08 PM.
  • The WisTransPortal database was unstable 9/14/2016 starting around 1:50 PM. Applications started having problems shortly after 2:30 and then most applications became unavailable by 3:00. The server was restarted and by 3:12 PM, all applications were back. We apologize for the inconvenience and will continue to investigate the issue.
  • Around 12:50 PM on Wednesday, August 31st, VSPOC application locked up all of WisTransPortal. Services was restored by 12:57 PM.
  • WisTransPortal server maintenance scheduled for Saturday, August 27th, 2016 was completed.
  • WisTransPortal TEST server maintenance scheduled for Tuesday, August 23rd, 2016 was completed on Thursday, August 25th.
  • WisTransPortal had an unexpected outage on August 9th, starting around 6:20 AM. The database archival storage reached maximum size and locked up. There were some other issues related to the newest Oracle patch applied last maintenance window that made this a little tougher to track down. System was back up by 11:35 AM.
  • A critical database maintenance window was set for Thursday, August 4th. The WisTransPortal service window started at 8:00 PM and ended at 10:05 PM. The outage affected all web based systems including TIA, InterCAD, LINK, and WisLCS.

  • WisTransPortal server maintenance window scheduled for Saturday, July 23rd, was completed. A longer then expected outage for applications happened due to a Oracle 11g update.
  • WisTransPortal TEST server maintenance window is scheduled for Tuesday, July 19th, was completed Friday, July 22nd.
  • WisTransPortal server maintenance scheduled for Saturday, June 25th - Completed.
  • WisTransPortal TEST server maintenance scheduled for Thursday, June 23rd - Completed
  • WisTransPortal server maintenance scheduled for Sunday, May 22nd, 2016 - COMPLETED

    Web server was slow again coming up after a reboot around 7:45AM and wasn't functional till around 10:00AM. After applying some VMware firmware update to host servers, server is responding much faster. A manual defrag on the drive was also done on Sunday.

  • TEST server maintenance window was completed on Friday, May 20th, 2016.
  • The WisTransPortal had a outage this morning (5/19/2016) around 7:00AM. There was a temporarily lock on the database due to a large import process that was running. The issue was resolved around 8:10AM.
  • WisTransPortal server maintenance scheduled for Sunday, May 1st, 2016 - COMPLETED

    Web server was slow coming up and wasn't functional till around 10:30AM. Researching the issue.

  • TEST server maintenance window was completed on Thursday, April 28, 2016.
  • WisTransPortal maintenance completed on Wednesday April 13th. starting @ 10:00 PM. This maintenance was necessary for adding additional storage capacity to the WisTransPortal database server. Maintenance was completed by 2:20AM on April 14th.
  • WisTransPortal Web services were locked up on 3/17/2016 from 2:46 - 3:00 PM. Web services were said to be slow much of the afternoon and VSPOC locking up at 2:46 put it over. An application issue with VSPOC appears to be the issue. IT is looking into the resolution.
  • VSPOC was down from 7:44 AM on 3/11/2016 till 10:24 AM on 3/11/2016. IT is looking into the issue.
  • WisTransPortal server maintenance on Sunday, March 20, 2016 - COMPLETED
  • TEST server maintenance window was skipped on Tuesday, March 15, 2016.
  • After WisTransPortal maintenance on Saturday, February 20th, there were a few applications that experienced an extended outage due to the updates. VSPOC was down until Sunday, February 21st at 9:40 PM. InterCAD was not updating on public websites till after 10:00 AM on Sunday, February 21st. The scheduling service was down until around 8:30 AM on Sunday morning. This affected updates for TIA, WisTMP, and VSPOC. All applications are back online but further review will be done to help alleviate the problems with future updates.
  • WisTransPortal server maintenance on Saturday, February 20, 2016 - COMPLETED
  • maintenance window was completed on Friday, February 19, 2016. No production servers were affected
  • WisTransPortal server maintenance on Saturday, January 23, 2016 - COMPLETED
  • <
    2016
    Scheduled WisTransPortal Server Maintenance Days Completed Notes
    December 18 SkippedPLANNED: Servers patched up to Microsoft December Updates; Symantec updates; VMware patches
    November 13 Completed Servers patched up to Microsoft November Updates; Symantec updates; New Remote Desktop Services server configured
    October 23 Completed Servers patched up to Microsoft October Updates; Symantec updates; VMware patches
    September 25 CompletedServers patched up to Microsoft September Updates; Symantec updates
    August 27 CompletedServers patched up to Microsoft August Updates; Dell firmware on select servers; Java Updates; Symantec updates
    July 23 CompletedServers patched up to Microsoft July Updates; Dell firmware on select servers; Java Updates; Symantec updates; Oracle 11g Update
    June 25 CompletedServers patched up to Microsoft June Updates; VMware Host patches; VM Client Tools upgraded
    May 22 CompletedServers patched up to Microsoft May Updates; VMware Host patches; Java Updates; Apache Security Patch
    May 1 CompletedServers patched up to Microsoft April Updates; VM Server Host Memory Upgrades
    March 20 CompletedServers patched up to Microsoft March Updates; Dell firmware on select hardware
    February 20 CompletedServers patched up to Microsoft February Updates; Apache Update; Tomcat Patch
    January 23 CompletedServers patched up to Microsoft January Updates; VMware Host patches; Java Update

2 0 1 5

  • WisTransPortal server maintenance on Wednesday, December 19, 2015 - COMPLETED
  • WisTransPortal server maintenance on Wednesday, November 18, 2015 - COMPLETED
  • WisTransPortal server maintenance on Saturday, October 24, 2015 - COMPLETED
  • WisTransPortal server maintenance on Saturday, September 19, 2015 - COMPLETED
  • WisTransPortal server maintenance on Saturday, August 22, 2015 - COMPLETED @ 3:00 PM
    • GIS services had an issue after the windows patches were installed so some mapping services were not functional until Sunday 23 around 7PM.

  • On 8/3/2015 around 10:05 AM, WisTransPortal admin was notified that the MV4000 Crash Data Resources was not showing documents as being available. Network configurations on the server were shown as being incomplete and the matter was repaired by 10:39 AM. This is a permanent solution. The issue on the server may have affected intermittent outages reported on Friday 7/31.
  • WisTransPortal server maintenance on Saturday, July 25, 2015 - COMPLETED
  • Around 12:30 PM on July 9, WisTransPortal Applications failed to work. The applications included TIA, Community Maps, VSPOC and WRS. The problem was caused by some new rules set on the WisTransPortal domain for a new stack of servers being brought up. These rules conflicted with the current ones in place and blocked access. The new rules were removed. Outage was approximately 20 minutes.
  • WisTransPortal server maintenance on Saturday, June 27, 2015 - COMPLETED
  • WisTransPortal server maintenance on Saturday, May 16, 2015 - COMPLETED
  • WisTransPortal server maintenance on Saturday, April 25, 2015 - COMPLETED on main production servers
  • WisTransPortal server maintenance on Saturday, March 21, 2015 - COMPLETED
  • On March 11 starting around 12:00 AM, the WISLR crash map and other GIS maps on the WisTransPortal were unavailable. This was due to a network change that inadvertanly affected permissions in these applications. Map services were restored around 1:00 PM on 3/11.
  • WisTransPortal server maintenance on Saturday, February 21, 2015 - COMPLETED
  • BITS/DET network maintenance on Sunday morning, February 8th, @ 5:00 AM, was unsuccessful. A roleback was performed at 3:30 PM. WisTransPortal tested fully back online by 6:00 PM
  • WisTransPortal server maintenance on Saturday, January 24, 2015 - COMPLETED
  • A short maintenance on the WisTransPortal main web server was completed on the evening of 1/6/2015. An upgrade of the Apache Web server with the latest patched version of 2.2.x.
  • On Sunday January 4th starting at midnight UW Network Services performed network maintenance. All BAN ASAs were upgraded to a version to protect them from a known vulnerability. ASA5505s were rebooted at midnight.
  • WisTransPortal server maintenance on Saturday, January 3, 2015 - SKIPPED DUE TO INCLEMENT WEATHER.
  • 2015
    Scheduled WisTransPortal Server Maintenance Days Completed Notes
    December 19 CompletedServers patched up to Microsoft December Updates; Dell firmware on select servers; Dell SAN firmware
    November 18 CompletedServers patched up to Microsoft November Updates
    October 24 CompletedServers patched up to Microsoft October Updates; VMware vCenter upgrade; Java update 65
    September 19 CompletedServers patched up to Microsoft September Updates; Apache Web patch
    August 22 CompletedServers patched up to Microsoft August Updates; VMware Client Tools Updates; Java Update; Symantec EP Version Update
    July 25 CompletedServers patched up to Microsoft July Updates; VMware Client Tools Updates; Oracle 11g Patch 39; Java Update on some PRD servers
    June 27 CompletedServers patched up to Microsoft June Updates; VMware Updates; Oracle 11g Patch 38
    May 16 CompletedServers patched up to Microsoft May Updates; Dell firmware on select servers; Virtual NIC fix on some virtual servers; NAS firmware; Hard Disk firmware update on VM SAN
    April 25 CompletedMain production servers patched up to Microsoft April Updates
    March 21 CompletedSelect production servers patched up to Microsoft March Updates; Dell firmware on select servers; VMware patches and updates on Host servers; Virtual NIC fix on select virtual servers; Dell EqualLogic firmware (VM SAN and File SAN); VMWare Tools update on all VM servers
    February 21 CompletedServers patched up to Microsoft February Updates; Dell firmware on select servers; Java 7 Update 76; Dell EqualLogic firmware (NAS SAN)
    January 24 CompletedServers patched up to Microsoft January Updates; VMware patches and updates on Host servers (PVMH1 and PVMH2); Virtual NIC fix on virtual servers

2 0 1 4

  • WisTransPortal server maintenance on Saturday, November 15, 2014 - COMPLETED.
  • WisTransPortal server maintenance on Saturday, October 25, 2014 - COMPLETED.
  • SUNDAY, OCTOBER 19 - Disabled SSL 3.0 support on the WisTransPortal web server to safeguard against the recent "POODLE" attack vulnerability.
  • WisTransPortal maintenance for GIS servers only, Friday, September 19, 2014 - COMPLETED
  • WisTransPortal server maintenance on Saturday, September 20, 2014 - COMPLETED.
  • WisTransPortal maintenance for GIS servers only, Friday, August 22, 2014 - COMPLETED
  • WisTransPortal server maintenance on Saturday, August 23, 2014 - COMPLETED.
  • WisTransPortal server maintenance on Saturday, July 26, 2014 - COMPLETED.
  • On June 18th at 7:55am, there was an interruption to electrical power to the UW-Madison Dayton Street data center, which houses many of the enterprise IT services on campus. The services that are housed in this data center were down. This included the WisTransPortal. Power was restored at 3:30 p.m. and campus systems started to be restored immediately after. Most of WisTrasnportal was up by 1:05 AM on June 19th and was fully back online at 10:00 AM on June 19th
  • LINK Video hardware has moved from the STOC in Milwaukee to DoIT Datacenter at the UW Madison on Wednesday evening, May 21 starting at 8:00 PM. Since the move, there has been issues with some hardware that causes the video to stop occasionally. Technicians are working on the issue. [CLOSED 6/2/2014 11:00 AM]
  • WisTransPortal server maintenance scheduled for Saturday, May 17, 2014 - Completed.
  • WisTransPortal server maintenance scheduled for Saturday, April 19, 2014 - Completed.
  • WisTransPortal server maintenance scheduled for Saturday, March 22nd, 2014 - Completed.
  • At about 11 AM on March 13th, email alerts were no longer being sent out from the WisTransPortal including TIA. The issue was on CAE servers and was resolved by 6:00 PM on 3/13.
  • At about 11 AM on March 13th, CAD incidents were not showing up on production. The issue was on the WisDOT servers. Problem was resolved by 6:30 PM on 3/13.
  • After the planned maintenance window of February 22nd, the 511Projects data was unavailable. Actual outage for the 511Projects data was from 2/22 @ 9:00 AM till 10:35 AM on 2/23.
  • WisTransPortal server maintenance scheduled for Saturday, February 22nd, 2014 - Completed.
  • During the planned maintenance window of 12:00 PM - 2:00 PM on Wednesday, January 29th, the Southwest video was unavailable. Actual outage for the SW Link video was from 12:15 PM till 4PM on 1/29.
  • WisTransPortal server maintenance scheduled for Saturday, January 18, 2014 - Completed.
  • On January 5, 2014 12:00 AM - 8:00 AM, DoIT will be doing network maintenance that will affect most of UW Madison. Router/Switch Code upgrades have been cancelled so we don't anticipate any outages for the WisTransPortal but these types of maintenance may sometimes produce some unpredicted results so notification is being given here just in case. (DoIT Notifications - Maintenance Notification; Router/Switch Upgrade CANCELLED)

  • 2014
    Scheduled WisTransPortal Server Maintenance Days Completed Notes
    December 20Skippeda few servers received Windows patches but most canceled due to Inclement weather
    November 15 CompletedServers patched up to Microsoft November Updates; Dell firmware on select servers; DB2 Oracle 11.2.0.4.0 update + 10/2014 patches; Java 7 Update 71
    October 25 CompletedServers patched up to Microsoft October Updates; Symantec ver 12.1.5
    September 20 CompletedServers patched up to Microsoft September Updates; VMWare Tools updated on all VMs; Firmware updates on TOPS SAN switches; Firmware update on DOT switches
    August 23 CompletedServers patched up to Microsoft August Updates; Java 7 Update 67; VMHost updates; SEP 12.1 RU4 MP1b
    July 26 CompletedServers patched up to Microsoft June and July Updates; Dell firmware on select servers
    June 28 Skipped
    May 17 CompletedServers patched up to Microsoft May Updates; Java 7 update 55
    April 19 CompletedServers patched up to Microsoft April Updates; VMware upgrade; Symantec Upgrade; Dell firmware on select servers
    March 22 CompletedServers patched up to Microsoft March Updates; Oracle patch 11.2.0.3 Patch28; Java 7 Update 51
    February 22 CompletedServers patched up to Microsoft February Updates; Dell EqualLogic firmware; VMware Patches;
    January 18 CompletedServers patched up to Microsoft January Updates; Symantec (all Managed); Dell firmware on select servers;

2 0 1 3

  • SW LINK Video was noted to be down on Dec 31 at 11:20 AM. Not certain how long they were out but SW LINK server had CPU errors starting on Dec 30 at 11:55 PM. Server was rebooted by DOT and the Encoders were running by 3:00 PM.
  • TIA email notifications stopped going out on Dec 30, around 7:00 PM. It was noted that CAE server maintenance was taking place on Dec 30, between 8 and 10 PM. Email services were not going to work during this time. When services were brought back online, CAE inadvertently limited the access to the mail servers. It was repaired by CAE and services were back on Dec 31 at 10:30 AM.
  • WisTransPortal server maintenance on Saturday, December 21, 2013 - Completed on main production servers only.

  • ArcGIS Server for 511 Projects stopped working. Problem was resolved at 11:05 AM on December 2, 2013. Service may have been out since Saturday maintenance on November 30, 2013, but uncertain.

  • WisTransPortal server maintenance on Saturday, November 30, 2013 - Completed

  • The WisTransPortal Database server ran out of log file space on October 22, 2013 around 11:30 PM. This caused an outage for many web apps including TIA and was due to some uploads of new data. Issue was resolved by 3:45 AM on October 23, 2013.

  • WisTransPortal server maintenance on Saturday, October 19, 2013 - Completed

  • The College locked down their mail server on Monday, September 30, 2013 around 2:50 PM. This was done in order to restrict a recent occurrence of spam originating from the campus VPN. This had the affect of inadvertently blocking email from the WisTransPortal servers. In order to resolve the issue, the College hostmaster added the two (public and private) WisTransPortal subnets into their configuration. The outage was resolved at 7:15 PM.

  • WisTransPortal server maintenance on Saturday, August 24, 2013 - Completed

  • WisTransPortal server maintenance on Saturday, July 27, 2013 - Completed

  • WisTransPortal applications became unresponsive around 1:30 PM on Tuesday, July 9 (TIA, LCS, Crash). Applications were back up by 1:49 PM.

  • WisTransPortal server maintenance on Saturday, June 22, 2013 - Skipped

  • WisTransPortal server maintenance on Saturday, May 25, 2013 from 7:00 AM - 11:00 AM. Completed.

  • Around 3:00PM on Monday, April 29, the WisTransPortal database server lost communication with the Web Applications. Applications were down for about 30 Minutes. Technicians restored communication by 3:33 PM.

  • Intermittent timeout and long delay issues have been identified on WisTransPortal Web applications as early as 12/17/2012 related to Java. A possible fix has been put in place and technicians watching results.(updated 4/24/2013). This issues is believed to be resolved (5/30/2013)

  • WisTransPortal server maintenance on Saturday, April 20, 2013 from 7:00 AM - 11:00 AM.

  • Issue on one of the WisTransPortal servers disabled some applications around 10:45 AM on 04/11/2013. Issue has been resolved with about a 10min outage affecting V-SPOC, WSR, InterCAD, and a few others.

  • WisTransPortal server maintenance took place Saturday, March 23, 2013 from 7:00 AM - 11:00 AM.

  • The WisTransPortal locked up on Tuesday, March 12, 2013 at around 12:30 PM. Tomcat was restarted on one of the servers to solve the problem. TIA, WISLCS, and Crash Data was affected. Outage was about 20 minutes.

  • WisTransPortal maintenance took place Saturday February 23, from 7:00 AM - 12:15 AM. Time was extended due to some non-production servers having some cleanup done and then not coming up right away. WisTransPortal Applications were not affected after 11:00 AM.

  • ** C A N C E L E D ** System maintenance window scheduled for Saturday January 19, from 7:00 AM - 10:30 AM.

  • ** C A N C E L E D ** SW Link server is tentatively scheduled to be moved in January 2013 from Madison to Milwaukee. This will be a half day outage for SW LINK video.


    2013
    Scheduled WisTransPortal Server Maintenance Days Completed Notes
    December 21 CompletedMain production servers patched up to Microsoft December Updates; Symantec;
    November 30 CompletedServers patched up to Microsoft November Updates; Symantec; Dell firmware on select servers;
    October 19 CompletedServers patched up to Microsoft October Updates; Symantec;
    September 14 CompletedServers patched up to Microsoft September Updates; Symantec; Dell firmware on select servers; Java 7 Update 40
    August 24 CompletedServers patched up to Microsoft August Updates; Symantec; Dell firmware on select servers; EqualLogic Firmware and Software updated
    July 27 CompletedServers patched up to Microsoft July Updates; Symantec; Dell firmware on select servers; DNS/DC replacement
    June 22 SkippedWisTransPortal services needed for road closures due to heavy rains
    May 25 CompletedServers patched up to Microsoft May Updates; Symantec
    April 20 CompletedServers patched up to Microsoft April Updates; Java versions updated (No upgrades); Firmware on select servers; Symantec
    March 23 CompletedPublic servers patched up to Microsoft March Updates; WEB1 gets Java 7; Symantec
    February 23 CompletedServers patched up to Microsoft February Updates; All servers have minimum IE8; Java 7 testing; Symantec
    January 19 SkippedWisTransPortal closure services needed concerning work being done around the Greenfield bridge; Snow days

2 0 1 2

  • Storm Reports was down from 11:54 PM on 12/26 till 9:11 AM on 12/27. Issue has been resolved.

  • WisTransPortal server maintenance was done on Saturday, December 22, 2012 from 7:00 AM - 11:00 AM. Services have been restored.

  • Circuit rebuilds between TOPS Lab, Hillfarms, and STOC were completed around 11:00 AM on December 21, 2012. Service is completed. iNOC Ticket# 79154

  • Scheduled ITSNET Fiber work was done on December 8, 2012. The outage was from 3:20 AM till 6:10 AM. The following applications were affected:
  • InterCAD
  • 511 XML
  • ** C A N C E L E D ** Scheduled ITSNET Fiber work to be done November 16, starting from 10:00 PM till November 17 ending at 6:00 AM. The following services will be affected:
  • LINK Video
  • InterCAD
  • 511 XML
  • ** C A N C E L E D ** Next system maintenance window scheduled for Saturday November 24, from 7:00 AM - 10:30 AM.

  • The WisTransPortal was having intermittent connection issues the morning of November 7, 2012 till 10:00 AM. Services have been restored. Fix was a WinSock patch.
  • The WisTransPortal was having intermittent connection issues from around 7:00 PM, October 30, 2012 till 5:05 AM on October 31, 2012. Services have been restored but cause is still undetermined.
  • The WisTransPortal was unavailable between 7:00 AM - 10:45 AM October 20, 2012 to install Windows Updates and Patches. Service has been restored.
  • SW LINK video was reported down @ 3:51 PM on Thursday September 27, 2012. Services was restored at 9:25 PM.

  • The WisTransPortal was unavailable between 7:00 AM - 10:45 AM September 23, 2012 to install Windows Updates and Patches. Service has been restored.

  • The WisTransPortal was unavailable on Wednesday, August 15, between 7:00 PM and Midnight. Equipment was being relocated to the DoIT Data Center. Service was restored at 1:00 AM on August 16.

  • The WisTransPortal was unavailable Monday, August 13, between 9:30 AM and 2:00 PM when network was switched over to new equipment. Service has been restored

  • The WisTransPortal was unavailable between 7:00 AM - 11:30 AM July 28, 2012 to perform hardware maintenance and install Windows Updates and Patches. Service has been restored.

  • The WisTransPortal was unavailable between 7:00 AM - 10:30 AM June 23, 2012 to perform hardware maintenance and install Windows Updates and Patches. Service has been restored.

    2 0 0 9

  • The WisTransPortal was unavailable between 6:00 AM - 9:30 PM September 26, 2009 due to an unexpected issue with a priority upgrade to one of the webserver components. Service has been restored.

    2 0 0 7

  • The WisTransPortal was unavailable between 8:00 AM - 4:00 PM June 24, 2007 to perform hardware maintenance on the webserver and apply an Oracle security patch. Service has been restored.

  • The WisTransPortal was temporarily unavailable between 5:12 AM - 6:05 AM May 24, 2007 due to unexpected system issues after installing a Windows software patch. Service has been restored.

  • The WisTransPortal was temporarily unavailable between 8:00 AM - 10:00 AM May 7, 2007 due to unexpected network issues. Service has been restored.

  • The WisTransPortal Oracle database was offline between 11:00 PM 3/10/2007 and 1:30 AM 3/11/2007 to apply DST and security maintenance patches.

  • There was a brief interruption in service between 9:00 AM - 10:00 AM Sunday, February 4, 2007 to upgrade the webserver software. The process completed successfully.

  • The WisTransPortal was offline briefly between 3:00 - 3:30 PM Friday, January 25, 2007, while a temporary hardware device was installed to address the intermittent hardware failure reported on January 11, 2007.

  • The WisTransPortal system lost network connectivity from approximately 1:30 AM - 5:00 AM Monday, January 18, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

  • The WisTransPortal system lost network connectivity from approximately 6:45 PM - 8:45 PM Monday, January 15, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

  • The WisTransPortal system lost network connectivity from approximately 5:30 PM - 9:30 PM Thursday, January 11, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

  • The WisTransPortal system lost network connectivity from approximately 8:00 AM - 9:30 AM Thursday, January 11, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

  • The WisTransPortal system lost network connectivity from 9:15 PM - 10:30 PM Tuesday, January 9, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

  • The WisTransPortal system lost network connectivity from approximately 3:00 PM - 4:30 PM Thursday, January 4, 2007. This outage affected access to all internet applications including the LINK video sharing system. Service has now been restored. We apologize for any inconvenience.

2 0 0 6

  • As of 10/30/2006, the WisTransPortal login form has been replaced with the pre-existing pop-up dialog. The login information (username and password) has not changed. This change was implemented in order to address several performance issues that were discovered during the initial testing phase.

  • The WisTransPortal system was offline from 9:43 PM 10/16/2006 - 5:35 AM 10/17/2006. Service has since been restored. We apologize for any inconvenience.

  • A maintenance window has been scheduled between the hours of 10:00 PM 10/7/2006 to 12:00 AM 10/8/2006 to upgrade the WisTransPortal webserver. All services, including WisDOT Traffic Video (LINK), the Crash Data export tool, and VSPOC, will be unavailable during that time.




About | Contact | Help | TOPS Lab


Copyright © 2019. Last Modified Sun, 03 Oct 2021, 03:33:27 PM. Questions or comments? transportal@topslab.wisc.edu.