If you're a Cisco Guy (or Girl, I'm not biased; both sexes are welcome to the Cisco Pain Train), you'll likely have come across a need for Policy Based Routing (PBR) to change the normal Destination IP-based logic of Network Routing to instead be a Source-IP or Interface-based logic, for some purpose of bypass. On Cisco gear, this is normally achieved via Route Maps or Route Policy Language (RPL) matching Access Control Lists (ACLs), Prefix Lists, Interfaces or some other attribute to match the desired traffic flows to "PBR" away from normal Routing.

Junos - or specifically Filter-based Forwarding (FBF) - takes a very different approach to achieving this, which may take your "Cisco brain" a bit of context-switching to adapt to, namely:

• PBR (FBF) is performed within a new, seperate, Routing Instance (VR, Virtual Router, or VRF)
  • So all entry (ingress) / exit (egress) interfaces will need to be "imported" to also "attach" to this PBR-specific VRF
• FBF is applied inbound on the Source-facing interface
  • So normal, non-PBR traffic flow still needs to be explicitly allowed through (otherwise it'll not just not be PBR'd, it'll be blocked from flowing entirely...)

Worked example

A worked example probably helps here, so suppose we have the following:

• Zscaler Internet Access (or equivalent Security Service Edge [SSE] or Cloud-based Internet Firewall/Proxy)
  • Traffic is forced into Zscaler via a GRE or IPsec Tunnel which runs atop the Internet into the closest/selected Zscaler Data Centre PoP (Point of Presence)
  • Zscaler (fake) Data Centre GRE Endpoint 165.225.99.99/32 acts as the far-side GRE Tunnel IP Endpoint/Peer
• 1 Gb Direct Internet Access (DIA) from BigNetCo with Private Allocated (PA) 192.0.2.0/24 IPv4 Address Block, effectively "leased" to Your Company
  • Static IPv4 /30 Handoff/Linknet 203.0.113.0/30 between Your Company and BigNetCo (out of BigNetCo's IPv4 Address Space)
• Default Route on your Juniper SRX towards VRF "Zscaler-Internet"
  • GRE Tunnel is transited via Underlay VRF Internet, but forms an extension of Overlay VRF Zscaler-Internet - that is, Default Route on LAN-side of Firewall forwards traffic (following the Default) via VRF Zscaler-Internet into the GRE Tunnel to Zscaler
  • Underlay VRF "Internet" only really exists to transit the GRE Tunnel itself (i.e. has a Static Route via next-hop 203.0.113.1 BigNetCo for GRE Peer Endpoint 165.225.99.99/32 only)
• VRF Prod exists as the only internal-facing DMZ/Security Zone, and routes the whole 10.0.0.0/8 Your Company Summary IPv4
  • Of this, most of 10.0.0.0/8 should have "Normal Internet access" - that is to say, flow via the GRE Tunnel to be scrubbed by Zscaler (i.e. appear to the Internet as a 165.x.x.x Zscaler Source IP)
  • Apart from "Direct Internet access" Subnet 10.2.0.0/24 - which (for reasons) needs to bypass Zscaler inspection and go direct to the Internet (i.e. appear to the Internet as a 192.0.2.0/24 Source IP)
  • And only then if they are going to HTTPS destinations; if they try and go to, say, HTTP or DNS, we want to send that via Zscaler inspection as per `"Normal Internet access"``

Topology

A picture paints a thousand words, so that summates as:

Configuration

Let's look at the raw config first, and then deconstruct it down to what this is doing. For brevity I've omitted the GRE Tunnel configuration, but if you're interested I suggest you read about Juniper SRX Overlay and Underlay VRF-seperated GRE Tunnels.

Ditto, in practice you'd want to Source NAT (SNAT) the traffic to a Public IPv4 to make it routable on the Internet, but again, for brevity I'm omitting that and focusing only on the PBR-required configuration itself - as you might want to adopt similar for Private-to-Private PBR requirements.

set routing-instances Internet-Bypass-ZIA instance-type forwarding
set routing-instances Internet-Bypass-ZIA routing-options static route 0.0.0.0/0 next-hop 203.0.113.1
set routing-instances Internet-Bypass-ZIA routing-options instance-import POL_EXPORT_FROM_Prod-VRF_TO_Internet-Bypass-ZIA
set routing-instances Internet-Bypass-ZIA routing-options instance-import POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA
!
set firewall family inet filter PBR-BYPASS-ZSCALER-ZIA term PBR-DIRECT-INTERNET from source-address 10.2.0.0/24
set firewall family inet filter PBR-BYPASS-ZSCALER-ZIA term PBR-DIRECT-INTERNET from destination-address 0.0.0.0/0
set firewall family inet filter PBR-BYPASS-ZSCALER-ZIA term PBR-DIRECT-INTERNET from destination-port https
set firewall family inet filter PBR-BYPASS-ZSCALER-ZIA term PBR-DIRECT-INTERNET then routing-instance Internet-Bypass-ZIA
set firewall family inet filter PBR-BYPASS-ZSCALER-ZIA term term-999 then accept
!
set policy-options prefix-list PFX_FROM_Prod_TO_Internet-Bypass-ZIA 10.99.99.0/30
set policy-options prefix-list PFX_FROM_Prod_TO_Internet-Bypass-ZIA 10.0.0.0/8
set policy-options policy-statement POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA term 10 from instance Prod
set policy-options policy-statement POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA term 10 from prefix-list PFX_FROM_Prod_TO_Internet-Bypass-ZIA
set policy-options policy-statement POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA term 10 then accept
set policy-options policy-statement POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA term 999 from instance Prod
set policy-options policy-statement POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA term 999 then reject
!
set policy-options prefix-list PFX_FROM_Internet_TO_Internet-Bypass-ZIA 203.0.113.0/30
set policy-options policy-statement POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA term 10 from instance Internet
set policy-options policy-statement POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA term 10 from prefix-list PFX_FROM_Internet_TO_Internet-Bypass-ZIA
set policy-options policy-statement POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA term 10 then accept
set policy-options policy-statement POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA term 999 from instance Internet
set policy-options policy-statement POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA term 999 then reject
!
set interfaces xe-0/0/1 unit 10 family inet filter input PBR-BYPASS-ZSCALER-ZIA

Break it down now y'all

That's quite a chunk to digest, so let's break it into chunks of what it achieves, using each exclamation mark (!) as a section divider:

1. Create a new VRF Internet-Bypass-ZIA for the PBR to occur within; map the Internet-facing WAN and Prod-facing LAN Interfaces and next-hop Prefixes into it, and set a Static Default Route out of it via the Internet-facing WAN interface
2. Define a FBF/PBR policy called PBR-BYPASS-ZSCALER-ZIA which has one term/policy statement called PBR-DIRECT-INTERNET that matches HTTPS requests from only the "Direct Internet access" Subnet 10.2.0.0/24, and bypasses (does not PBR) any other flows (i.e. leaves them as-was to flow via Zscaler Internet as if the PBR configuration didn't exist)
3. Create a Routing Policy POL_EXPORT_FROM_Prod_TO_Internet-Bypass-ZIA to inter-VRF the LAN-facing 10.99.99.0/30 Handoff/Linknet and associated next-hop 10.0.0.0/8 Route from the Prod VRF into the PBR Internet-Bypass-ZIA VRF
4. Create a Routing Policy POL_EXPORT_FROM_Internet_TO_Internet-Bypass-ZIA to inter-VRF the WAN-facing 203.0.113.0/30 Handoff/Linknet from the Internet VRF into the PBR Internet-Bypass-ZIA VRF
5. Tie it all together by applying the FBF/PBR policy PBR-BYPASS-ZSCALER-ZIA to ingress LAN-facing interface xe-0/0/1.10
   1. Note if we changed the FBF/PBR policy PBR-BYPASS-ZSCALER-ZIA term-999 line from "...then accept" to "...then reject" - or just did not have that line - then any non-PBR'd traffic would not just not be PBR'd, it'd be dropped entirely - so be very careful that the PBR/FBF policy is 100% correct before you apply it to the ingress interface

One hop this time

Hopefully this has been a useful jaunt through the differences Junos SRX uses to perform what we Cisco folk would know as PBR, and shows how it works in practice, effectively using a "third" VRF - in our example - which acts as a PBR Container to glue it all together. In practice this is nice as it keeps it obvious which flows/Source IPs/Destination IPs are being PBR'd and which aren't, however it can be confusing to wrap your head around. I also found a few gotchas if you forget/mismatch the Linknets to "import" the WAN/LAN interfaces into what is effectively a "foreign VRF" from their respective "Prod" or "Internet" native VRFs.

Are you used to Cisco IOS kit and trying to make a GRE Tunnel extend an Overlay VRF (or Routing Instance), using Junos SRX kit, and getting weird errors like this:

(errno=1000) create nsp tunnel failed 1
(errno=1000) tunnel session add(gr-0/0/0) failed

Then read on, dear friend - this post's for you.

Diagram time

This is what we're trying to achieve, namely:

• Underlay Network
  • MPLS IP VPN with own BGP/IGP routing setup
  • Two Juniper SRX Firewalls in two locations, connected to said MPLS IP VPN
  • Underlying Global Table or VRF "Prod" that those Firewalls connect to
  • Each Firewall has 1x 10 Gig Ethernet connection, Xe-0/0/1, connected to the MPLS IP VPN
• Overlay Network
  • Routing Instance (VRF/Routing Table) called "Other" present on each Juniper SRX Firewall
  • We want to join together VRF "Other" on each SRX Firewall to each other, using a GRE Tunnel Gr-0/0/0.1 to extend them
  • We'll spin up a Routed /30 p2p atop this GRE Tunnel, so we can Static Route to it, each side, to get traffic from SiteA to SiteB

The ultimate goal is for Gr-0/0/0.1 on each SRX to be able to talk to each other's 192.168.99.x IP directly, over the GRE Tunnel (which gets transited atop the Underlay MPLS IP VPN, VRF "Prod").

What your Cisco mind tells you might work

This isn't your first GRE rodeo, you've played this game before, and done something similar on Cisco IOS kit, like this:

interface Tunnel99
 description GRE Tunnel to SiteB Tun99
 ip vrf forwarding Other
 ip address 192.168.99.1 255.255.255.252
 ip mtu 1476
 tunnel source 10.0.0.7
 tunnel destination 10.1.0.98

That worked well for you, so you reckon some config conversion like this might bear some fruit:

set interfaces gr-0/0/0 unit 1 description "GRE Tunnel to SiteB Gr-0/0/0.1"
set interfaces gr-0/0/0 unit 1 tunnel source 10.0.0.7
set interfaces gr-0/0/0 unit 1 tunnel destination 10.1.0.98
set interfaces gr-0/0/0 unit 1 tunnel routing-instance destination Other
set interfaces gr-0/0/0 unit 1 family inet mtu 1476
set interfaces gr-0/0/0 unit 1 family inet address 192.168.99.1/30
!
set routing-instances Other interface gr-0/0/0.1

That'll do it, right?

Nope.

What the Juniper SRX errors to fight back

So you try your ping 192.168.99.2 routing-instance Other, it fails miserably; then you go down to checking the Underlay Tunnel Source can ping the Tunnel Destination:

PING 10.1.0.98: 56 data bytes 64 bytes from 10.0.0.7:
icmp_seq=0 ttl=122 time=5.661 ms 64 bytes from 10.1.0.98
icmp_seq=1 ttl=122 time=6.619 ms 64 bytes from 10.1.0.98

Hmm, that's all fine, Alright, Juniper, what about a little show log messages | last 10 then, eh?

Aug 21 23:00:03 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed

Huh, what's that all about?

Hunting the Interwebs for clues

Time to invoke Dr Google then, and off we find this similarly afflicted person with an SRX1400, with this key scrap of thinking material:

This error is when you need the static route if gr-0/0/0 and egress interface are in two different routing-instances. So you need to point the static route of the gr tunnel to the table that have the external interface.

It's not really sinking in, so you re-read it a few times; why does Junos care if my Gr-0/0/0.1 interface is in a different Routing Instance (VRF) to the Underlay, that's the point of the ...tunnel routing-instance destination Other command, right, to tell it to set the destination of the Tunnel Endpoint into another Routing Table, right?

Nope.

In Juniper land, Destination Sources you!

It's probably best if I show you the fix first, then this might click - here is the fix to suddenly make the Tunnel spring into life:

set routing-instances Other routing-options static route 10.1.0.98/32 next-table Prod.inet.0

Note1: You'll need the other IP, for SiteA (10.0.0.7) as this inter-VRF Static Route for the other SiteB Firewall

Note2: You'll need to add that ".inet.0" to your Routing Instance name, because Junos and consistency don't mix.

So if you're like me, you're looking at that bemused, and saying to yourself: "But, but... I set the Tunnel Destination to look in a different VRF, right, otherwise what exactly is the point of that routing-instance destination command all about?".

Well, I'll tell you what I reckon it's to do, I think it's to set the source of the Tunnel (i.e. what Cisco would call the "Tunnel Source") to use the Underlay (Prod VRF) to form; not, as the fecking word in the command reads, the Tunnel Destination. Hence, as there is no command that sets the destination, it assumes the Tunnel Destination is in the same VRF as the Tunnel itself, so you have to set an inter-VRF Static Route to tell it to go into the Underlay to form the GRE Tunnel.

It's the only way I can reconcile it in my head, with you needing to add he destination route/prefix as well as that ...tunnel routing-instance destination... command.

Conclusion

So there you go, in Junos land, this magic badly-named combo does what the sensibly-named tunnel source and tunnel destination combo does in Cisco IOS:

set interfaces gr-0/0/0 unit 1 tunnel source 10.0.0.7 #Note this is in VRF Prod underlay
set interfaces gr-0/0/0 unit 1 tunnel destination 10.1.0.98 #Note this is in VRF Prod underlay
set interfaces gr-0/0/0 unit 1 tunnel routing-instance destination Prod #Note this sets the VRF for the Tunnel Source
set routing-instances Other routing-options static route 10.1.0.98/32 next-table Prod.inet.0 #Note this inter-VRFs the Tunnel Destination

Mind you, in Juniper land opposites are clearly true, so maybe I should call this section the Introduction instead, seeing as it's at the end of the blog post.

If only Juniper sold mineral water, it'd be bottled at destination instead.

Sorry, I'll start now. Wait, do I mean stop...