TPG Community

theexecutioner · ‎07-04-2019

Hi folks,

fwiw a support ticket has been raised today, but thought I'd post to ask if anyone knows what is going on.

Background: I'm a software engineer with some network engineering background (enough to be dangerous ) and keep a close eye on latency on my network (being an avid gamer).

In the past week, something has been wrong - initially I thought it might be game bugs which are incorrectly determining latency into various AWS regions, but today I did further investigation, and have to put the blame squarely into TPG's lap, or the lap of one of the downstream partners - TPG need to fix this:

Below is a current trace of the route my packets are taking to reach an EC2 AWS gaming instance in Singapore (SEA - South East Asia - region). Notice the highlights... it's routing me through Los Angeles in the USA!!

To further diagnose, I activated my NordVPN VPN endpoint to a Melbourne location, and the trace went through the undersea cable via Perth directly to Singapore, so it's isolated to TPG based routing.

Note that in both cases, the EC2 server does not respond to ICMP packet latency requests, and so the trace drops one server back (but still in Singapore if one does a geolocation on the IP address).

fwiw, it only seems to affect ICMP packets - once I have established a connection to the Singapore endpoint and drop the VPN, all routing appears to be optimal (sub-100ms latency) - but ICMP ping requests are not routing correctly...

Any idea how this can be resolved?

(TPG routing):

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 1 ms <1 ms <1 ms 192.168.0.1
2 2 ms 1 ms 1 ms 192-168-1-1.tpgi.com.au [192.168.1.1]
3 9 ms 9 ms 9 ms 10-20-25-29.tpgi.com.au [10.20.25.29]
4 9 ms 11 ms 10 ms 203-219-155-130.tpgi.com.au [203.219.155.130]
5 25 ms 22 ms 23 ms syd-gls-har-crt2-be-10.tpgi.com.au [202.7.173.17]
6 27 ms 22 ms 32 ms 203-221-3-67.tpgi.com.au [203.221.3.67]
7 169 ms 168 ms 169 ms las-b24-link.telia.net [213.248.95.232]
8 184 ms 188 ms 185 ms tata-ic-333371-las-b24.c.telia.net [80.239.128.215]
9 360 ms 361 ms 359 ms if-ae-2-2.tcore2.lvw-los-angeles.as6453.net [66.110.59.2]
10 527 ms 288 ms 288 ms if-et-53-2.hcore2.kv8-chiba.as6453.net [64.86.252.57]
11 374 ms 379 ms 373 ms if-ae-23-2.tcore1.svw-singapore.as6453.net [180.87.67.32]
12 362 ms 365 ms 361 ms 180.87.12.26
13 * * * Request timed out.
14 * * * Request timed out.
15 374 ms 384 ms 371 ms 52.93.10.57
16 370 ms 368 ms 369 ms 52.93.11.85
17 369 ms 368 ms 369 ms 52.93.11.94
18 362 ms 362 ms 362 ms 52.93.9.185
19 370 ms 369 ms 368 ms 52.93.10.101

(NordVPN routing):

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 10 ms 9 ms 12 ms 10.8.8.1
2 11 ms 15 ms 11 ms 144.48.37.105
3 10 ms 10 ms 11 ms xe-4-0-0.cr2.me1.vic.au.coloau.net.au [103.52.117.104]
4 93 ms 92 ms 92 ms xe-4-0-2.cr1.p1.wa.au.coloau.net.au [45.112.247.163]
5 96 ms 93 ms 92 ms silicondesertinc.te0-6-1-18.br04.sin02.pccwbtn.net [63.217.25.194]
6 98 ms 92 ms 95 ms te0-6-1-18.br04.sin02.pccwbtn.net [63.217.25.193]
7 95 ms 92 ms 95 ms 63-218-107-42.static.pccwglobal.net [63.218.107.42]
8 * * * Request timed out.
9 * * * Request timed out.
10 153 ms 164 ms 155 ms 52.93.10.9
11 * * * Request timed out.
12 149 ms 149 ms 149 ms 52.93.11.8
13 148 ms 149 ms 148 ms 52.93.8.119
14 148 ms 147 ms 155 ms 203.83.223.33

theexecutioner · ‎02-05-2019

Firstly, thx to all who have taken an interest in this thread, especially the moderators.

I’m pasting the body of the reply from the technical team below for anyone who wishes to know the final outcome. Ultimately, it appears that the routes via Perth and the lower latency undersea cables for these particular Singaporean servers are not being advertised to TPG by their available networking peers, and instead the routes take TPG customers via LA - causing the higher latency.

I’ll not comment further beyond acknowledging that I have agreed to close the ticket, on the basis that TPG is unable to make the service any better at this time. I am disappointed with the outcome, especially when competitors in this space don’t seem to suffer from the same routing issues (refer my previous post regarding the lookingglass services)... hopefully sometime in the future peering arrangements can be negotiated that rectify this.

Cheers, and have a good day one and all.
————————- Body of final TPG reply follows::
Thank you for choosing TPG as your service provider.
We note that you reported Latency issue with your internet service specifically when reaching SEA Servers.
This concern has been raised to our Network Team's attention.
After thorough investigation, we have come to the conclusion that the route in-place is the optimal route currently available for your service considering Network traffic and available sources.
TPG is unable to learn route from our Asia Peer thus the connection routes to Los Angeles Server which is advertised by our peer.
To add, TPG cannot control any routing from the ISP where this server is connected. It will depend if they want TPG to learn the route via Asia.
Future changes may come into place however we do not have definite date at the moment.
We have concluded our investigation of the fault you have reported, all equipment within the TPG network is working normally.
If you choose to remain a TPG customer please be aware that at this stage no further action can be taken to improve the quality of your service. Additionally you will continue to be charged as normal for the service provided.

View solution in original post

theexecutioner · ‎04-09-2019

You censored my post regarding latency being so much better now that I've churned?

Why?

Bitter pill not to be swallowed?

View solution in original post

theexecutioner · ‎10-04-2019

Nice to see TPG responding to the more technical issues ... cheers! /sarcasm

Anonymous · ‎10-04-2019

Hi @theexecutioner

Thanks for raising this to us. It is pretty obvious on your post that there is something wrong with the routing causing latency issue. This issue needs to be raised to our Engineering Team for further test and investigation.

We tried to use your community details to pull up the account unfortunately it shows multiple matching result.

Feel free to PM us your account details (Username/Customer ID together with the address on file).

In case you need a reference:

How do I private message (PM) in the community

Cheers!

theexecutioner · ‎11-04-2019

Time's moved forward ... and the eng. team is apparently investigating. I'll post once I have an outcome. fwiw, Optus routes are fine ... so are NordVPN's ... but not TPG's .... so hopefully the eng. team can fix or trace to a downstream partner and get them to fix ... Cheers.

Anonymous · ‎11-04-2019

We are glad to know that this issue is now being handled by our Engineering Team @theexecutioner. We'll keep an eye on this thread. Let us know how the case progress will go.

Cheers!

theexecutioner · ‎15-04-2019

The Engineering team have fixed it apparently ...

Does that look fixed to you....? Where did these bozo's learn to test...?

tracert ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com

Tracing route to ec2-13-251-110-6.ap-southeast-1.compute.amazonaws.com [13.251.110.6]
over a maximum of 30 hops:

1 3 ms <1 ms <1 ms 192.168.0.1
2 2 ms 1 ms 1 ms 192-168-1-1.tpgi.com.au [192.168.1.1]
3 11 ms 14 ms 13 ms nme-sot-dry-bras4-lo-20.tpgi.com.au [10.20.20.167]
4 16 ms 10 ms 12 ms nme-apt-bur-crt1-be10.tpgi.com.au [203.219.155.129]
5 32 ms 31 ms 30 ms syd-gls-har-crt1-be-10.tpgi.com.au [202.7.171.173]
6 32 ms 33 ms 28 ms syd-gls-har-int1-he0-3-0-2.tpgi.com.au [203.221.3.3]
7 174 ms 177 ms 177 ms las-b24-link.telia.net [213.248.95.232]
8 184 ms 185 ms 187 ms tata-ic-333371-las-b24.c.telia.net [80.239.128.215]
9 264 ms 266 ms 292 ms if-ae-2-2.tcore2.lvw-los-angeles.as6453.net [66.110.59.2]
10 272 ms 274 ms 277 ms if-ae-7-2.tcore2.svw-singapore.as6453.net [180.87.15.25]
11 265 ms 266 ms 267 ms 180.87.15.206

BasilDV · ‎16-04-2019

Hi @theexecutioner,

Thanks for the update.

Please shoot me a PM with your TPG username or customer ID number. We'd like to look into it further.

Kind regards,

BasilDV

theexecutioner · ‎18-04-2019

My original ticket has now been reopened. Anything you can do to help facilitate would be appreciated ... cheers.

Anonymous · ‎18-04-2019

Hi @theexecutioner,

Thanks for notifying us. We will chase this up with our Engineering Team. Based on the latest log on file, we've seen that you have been in contact with one of our Engineers and advised that the Escalated Fault was re-opened, updates will be provided once update is available via phone call or SMS.

Let us know should you require further assistance.

Cheers!

theexecutioner · ‎19-04-2019

Had a callback from the TPG team today ... "We don't have control over the routing..."

REALLY??

This is the answer from the network team? My goodness. While I accept that the TPG team can't directly make change if partners are incorrectly routing (assuming your primary routing table is correctly advertising optimal routes and it's being overriden downstream) I *absolutely* expect TPG to interact with industry partners to correct these type of problems downstream.

EDIT: To reiterate - I have had this same trace executed on an Optus cable connection and also when routed through NordVPN into Melbourne - BOTH route correctly, whereas TPG do not.

I'm thinking this may need to be raised up through an Industry Ombudsman ticket if I keep getting these type of answers from Network Support.

The clock is now ticking. I'm getting fed up with the run around...

TPG Community

Routing to AWS EC2 instances in SEA (Singapore) from Melbourne - something wrong